LearningSlowly LearningSlowly - 15 days ago 11
Python Question

Multiprocessing list manager

I have a function that I wish to pass different input files to simultaneously. I am using the multiprocessing manager for the output list,

nodes
. I am defining this as
nodes = manager.list()


file_list = [file_1,file_2,file_3]

def function_x(file,nodes):
nodes.extend(some_data)
print(type(nodes))

if __name__ == "__main__":
manager = multiprocessing.Manager()
nodes = manager.list()
matches = partial(function_x,nodes)
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
pool.map(function_x,file_list)


Despite defining
nodes
as a list, I am getting the following error:
AttributeError: 'str' object has no attribute 'extend'


When I print the
nodes
type I get back
string
. Why is
nodes = manager.list()
not defining this properly?

Answer

The trouble is in matches = partial(function_x, nodes).
Here partial substitutes the first function_x argument (eg file) with nodes and you get filename (a string) in the second argument, hence the error.

So either swap the function_x arguments:

def function_x(nodes, filename):

or use keyword argument when constructing the partial:

matches = partial(function_x, nodes=nodes)
Comments