LearningSlowly LearningSlowly - 11 months ago 76
Python Question

Multiprocessing list manager

I have a function that I wish to pass different input files to simultaneously. I am using the multiprocessing manager for the output list,

. I am defining this as
nodes = manager.list()

file_list = [file_1,file_2,file_3]

def function_x(file,nodes):

if __name__ == "__main__":
manager = multiprocessing.Manager()
nodes = manager.list()
matches = partial(function_x,nodes)
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())

Despite defining
as a list, I am getting the following error:
AttributeError: 'str' object has no attribute 'extend'

When I print the
type I get back
. Why is
nodes = manager.list()
not defining this properly?

Answer Source

The trouble is in matches = partial(function_x, nodes).
Here partial substitutes the first function_x argument (eg file) with nodes and you get filename (a string) in the second argument, hence the error.

So either swap the function_x arguments:

def function_x(nodes, filename):

or use keyword argument when constructing the partial:

matches = partial(function_x, nodes=nodes)