mel mel - 6 months ago 34
Python Question

Multiprocess a function in python that got multiple parameters

I'm trying to use the multiprocessing library in python but I met some difficulties:

def request_solr(limit=10, offset=10):
# build my facets here using limit and offset
# request solr
return response.json()

def get_list_event_per_user_per_mpm(limit=100):
nb_unique_user = get_unique_user()
print "Unique user: ", nb_unique_user
processor_pool = multiprocessing.Pool(4)
offset = range(0, nb_unique_user, limit)
list_event_per_user =, offset)
return list_event_per_user

I'm not sure how to pass the second parameters into the function. How can I make it work. I've got the following error:

TypeError: 'dict' object is not callable


You need to use a lambda for this. The way you're doing it right now, it's trying to map the result of request_solr as a function with offset as the argument.

This should do the trick. x: request_solr(limit, x), offset)

Note, this only works in 3.x. In 2.x you need to create a function object. For example:

class RequestSolrCaller:
    def __init__(self, limit)
        self.limit = limit
    def __call__(self, offset)
        return request_solr(self.limit, offset), offset)