I've been using concurrent.futures as it has a simple interface and let user easily control the max number of threads/processes. However, it seems like concurrent.futures hides failed tasks and continue the main thread after all tasks finished/failed.
return (i + 's')
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
fs = [executor.submit(f, i ) for i in range(10)]
concurrent.futures.wait will ensure all the tasks completed, but it doesn't check success (something
return-ed) vs. failure (exception raised and not caught in worker function). To do that, you need to call
.result() on each
Future (which will cause it to either re-
raise the exception from the task, or produce the
return-ed value). There are other methods to check without actually raising in the main thread (e.g.
.result() is the most straightforward method.
If you want to make it re-
raise, the simplest approach is just to replace the
wait() call with:
for fut in concurrent.futures.as_completed(fs): fut.result()
which will process results as
Futures complete, and promptly
Exception if one occurred. Alternatively, you continue to use
wait so all tasks finish before you check for exceptions on any of them, then iterate over
fs directly and call
.result() on each.