There are three questions as possible duplicates (but too specific):
By answering this question all three other questions can be answered.
Hopefully I make myself clear:
Once I created an object in some process created by multiprocessing:
Example 1 (solved)
- How do I pass a reference to that object to an other process?
- (not so important) How do I make sure that this process does not die while I hold a reference?
from concurrent.futures import *
return lambda: v * v
if __name__ == '__main__':
with ThreadPoolExecutor(1) as e: # works with ThreadPoolExecutor
l = list(e.map(f, [1,2,3,4]))
print([g() for g in l]) # [1, 4, 9, 16]
returns an object with mutable state. This identical object should be accessible from other processes.
I have an object which has an open file and a lock - how do I grant access to other processes?
I do not want this specific error to not appear. Or a solution to this specific usecase. The solution should be general enough to just share unmovable objects between processes. The objects can potentially be created in any process. A solution that makes all objects movable and preserves identity can be good, too.
Any hints are welcome, any partial solution or code fragments that point at how to implement a solution are worth something. So we can create a solution together.
Here is an attempt
to solve this but without multiprocessing: https://github.com/niccokunzmann/pynet/blob/master/documentation/done/tools.rst
What you want the other processes to do with the references?
The references can be passed to any other process created with multiprocessing(duplicate 3). One can access attributes, call the reference. Accessed attibutes may or may not be proxies.
What's the problem with just using a proxy?
Maybe there is no problem but a challenge. My impression was that a proxy has a manager and that a manager has its own process and so the unserializable object must be serialized and transfered (partially solved with StacklessPython/fork).
Also there exist proxies for special objects - it is hard but not impossible to build a proxy for all objects (solvable).
Solution? - Proxy + Manager?
Eric Urban showed that serialization is not the problem. The real challenge is in Example2&3: the synchronization of state. My idea of a solution would be to create a special proxy class for a manager. This proxy class
- takes a constuctor for unserializable objects
- takes a serializable object and transfers it to the manager process.
- (problem) according to 1. the unserializable object must be created in the manager process.