Moinuddin Quadri Moinuddin Quadri - 15 days ago 7
Python Question

Running multiple asynchronous function and get the returned value of each function

I was trying to create a function which can run multiple processes asynchronous and will send the response. Since

multiprocessing.Process()
do not return the response, I thought of creating a function as:

from multiprocessing import Process

def async_call(func_list):
"""
Runs the list of function asynchronously.

:param func_list: Expects list of lists to be of format
[[func1, args1, kwargs1], [func2, args2, kwargs2], ...]
:return: List of output of the functions
[output1, output2, ...]
"""
response_list = []
def worker(function, f_args, f_kwargs, response_list):
"""
Runs the function and appends the output to list
"""
response = function(*f_args, **f_kwargs)
response_list.append(response)

processes = [Process(target=worker, args=(func, args, kwargs, response_list)) \
for func, args, kwargs in func_list]

for process in processes:
process.start()
for process in processes:
process.join()
return response_list


Within this function, I call
worker
asynchronously which accepts additional parameter as
list
. Since, lists are passed as reference, I thought I can append the response of actual function within the list. And
async_call
will return me the response of all the function.

But this is not behaving the way I expected. Value gets appended to the
list
within the
worker()
, but outside the worker
response_list
list remains empty.

Any idea what I am doing wrong? And, is there any alternative to achieve what I am doing?

Answer

You can't share objects directly across processes. You need to use one of the classes especially designed for communicating values, Queue and Pipe; see the documentation.

Comments