This is yet another question regarding
for i in range(int(1e6)):
if __name__ == "__main__":
queue = Queue(maxsize=-1)
pool = Pool(processes=12)
pool_obj_worker = pool.map_async(worker, gen_data(), chunksize=1)
print ('Lets run the workers...\n')
print ('\nAll processed successfully!') # I can see this quickly, so my jobs are done
print ('\nAll processed. Errors encountered!')
print (q.qsize()) # The size is right that means all workers have done their job
pool.join() # will get stuck here for long long time
print ('%d still to be processed' %
The issue here is that you are using an extra
Queue in your worker, other than the one fournished by
When the processes finish their work, they will all join the
FeederThread used in the
multiprocessing.Queue and these calls will hang (probably because all the threads call
join simultaneously and there can be some weird race conditions, it is not easy to investigate).
multiprocessing.util.log_to_stderr(10) permits to reveal that your processes hang while joining the queue feeder thread.
To solve your issue, you can either use
multiprocessing.SimpleQueue instead of
multiprocessing.Queue (no hang in join as there is no feeder thread) or try using the method
pool.unordered_imap which provides the same kind of behavior as what you seem to implement (give back an unordered generator containing the results returned by worker).