Mermoz Mermoz - 1 year ago 114
Python Question

Multiprocessing: How to use on a function defined in a class?

When I run something like:

from multiprocessing import Pool

p = Pool(5)
def f(x):
return x*x, [1,2,3])

it works fine. However, putting this as a function of a class:

class calculate(object):
def run(self):
def f(x):
return x*x

p = Pool()
return, [1,2,3])

cl = calculate()

Gives me the following error:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/sw/lib/python2.6/", line 532, in __bootstrap_inner
File "/sw/lib/python2.6/", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/sw/lib/python2.6/multiprocessing/", line 225, in _handle_tasks
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I've seen a post from Alex Martelli dealing with the same kind of problem, but it wasn't explicit enough.

Answer Source

I also was annoyed by restrictions on what sort of functions could accept. I wrote the following to circumvent this. It appears to work, even for recursive use of parmap.

from multiprocessing import Process, Pipe
from itertools import izip

def spawn(f):
    def fun(pipe,x):
    return fun

def parmap(f,X):
    pipe=[Pipe() for x in X]
    proc=[Process(target=spawn(f),args=(c,x)) for x,(p,c) in izip(X,pipe)]
    [p.start() for p in proc]
    [p.join() for p in proc]
    return [p.recv() for (p,c) in pipe]

if __name__ == '__main__':
    print parmap(lambda x:x**x,range(1,5))
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download