I have python code which reads data as stream (sys.stdin) and then perform some action for each line.
Now as volume of data is increasing, i want to split the task among threads and let them work in parallel.
Went through the docs and most of them suggest that threads need to poll (eg from Queue) to get task and work upon it. Here i need to push tasks to these threads.
Any idea/link where i can figure out how to do this ?
for line in sys.stdin:
//perform some action, which needs to be split among threads
//action is I/O-bound
from concurrent.futures import ThreadPoolExecutor import sys def some_action(line): pass # TODO: the actual task with ThreadPoolExecutor() as executor: for line in sys.stdin: future = executor.submit(some_action, line)