AlwaysSunny AlwaysSunny - 3 years ago 175
Python Question

How to use pool-specific functions while multiprocessing with python?

I'm using selenium and multiprocessing to spawn four different websites, and I want to run functions specific to the website the driver generated.

This is similar to my current code:

from multiprocessing import Pool
from selenium import webdriver

def gh(hosts):
driver = webdriver.Chrome(executable_path='./chromedriver')
driver.get(hosts)
html_source = driver.page_source
if 'ryan' in html_source:
print 'ryan'
doSomethingForRyan()
elif 'austin' in html_source:
print 'austin'
doSomethingForAustin()
elif 'travis' in html_source:
print 'travis'
doSomethingForTravis()
elif 'levi' in html_source:
print 'levi'
doSomethingForLevi()
else:
print '--NONE--'

if __name__ == '__main__':
p = Pool(4)

hosts = ["http://ryan.com", "https://www.austin.com", "http://levi.com", "http://travis.com"]
p.map(gh, hosts)


The result I'm getting is something like:
austin
austin
ryan
austin

Answer Source

EDIT - SOLVED

Instead of reading from driver.page_source, reading from driver.current_url makes sure that I can run website-specific functions.

if 'ryan' in driver.current_url:
    print 'ryan'
    doStuff()
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download