user3089927 user3089927 -4 years ago 182
Python Question

Multithreading in python 2.7

I am not sure how to do multithreading and after reading a few stackoverflow answers, I came up with this.
Note: Python 2.7

from multiprocessing.pool import ThreadPool as Pool
pool_size=10
pool=Pool(pool_size)




for region,directory_ids in direct_dict.iteritems():
for dir in directory_ids:
try:
async_result=pool.apply_async(describe_with_directory_workspaces,(region,dir,username))
result=async_result.get()
code=result[0]
content=result[1]
except Exception as e:
print "Some error happening"
print e

if code==0:
if content:
new_content.append(content)
else:
pass
else
return error_html(environ,start_response,content)


What I am trying to do here is calling "describe_with_directory_workspaces" with different parameters of region and directories and run it in parallel so that I get the data quickly in new content.Currently, it is going in series which is what giving slow performance to end user. Am I doing it right? or someother better way to do it? How can I confirm that I am getting the multithreading running as I expected it to?

Answer Source

You don't want to call async_result.get until you've queued all of your jobs, otherwise you will only allow one job to run at a time.

Try queueing all of your jobs first, then processing each result after they've all been queued. Something like:

results = []
for region, directory_ids in direct_dict.iteritems():
    for dir in directory_ids:
        result = pool.apply_async(describe_with_directory_workspaces,
                                  (region, dir, username))
        results.append(result)

for result in results:
    code, content = result.get()
    if code == 0:
        # ...

If you want to handle the results in an asynchronous callback, you can supply a callback argument to pool.apply_async as documented here:

for region, directory_ids in direct_dict.iteritems():
    for dir in directory_ids:
        def done_callback(result):
            pass  # process result...

        pool.apply_async(describe_with_directory_workspaces,
                         (region, dir, username),
                         done_callback)
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download