Mr. Bing Mr. Bing - 1 year ago 39
Python Question

Parallel execution of 2 seperate python scripts using subprocess from the main Python script

Using python, I want to start two subprocesses in parallel. One will start an HTTP Server, while other will start execution of another program(which is a python script generated by selenium IDE plugin to open firefox, navigate to a website and do some interactions). On the other hand, I want to stop execution of the first subprocess(the HTTP Server) when the second subprocess is finished executing.

The logic of my code is that the selenium script will open a website. The website will automatically make a few GET calls to my HTTP Server. After the selenium script is finished executing, the HTTP Server is supposed to be closed so that it can log all the captured requests in a file.

Here is my code:

class Myclass(object):


def startHTTPServer(self):
print "********HTTP Server started*********"
self.HTTPSERVERPROCESS=subprocess.Popen('python', \
shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
except Exception as e:
print "Exception captured while starting HTTP Server process: %s\n" % e

def startNavigatingFromBrowser(self):
print "********Opening firefox to start navigation*********"
process=subprocess.Popen('python /Users/akshay/PycharmProjects/SprinklrXSS/', \
shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except Exception as e:
print "Exception captured starting Browser Navigation process : %s\n" % e
if process.returncode==0:
print "HTTPSERVEPROCESS value: %s" % self.HTTPSERVERPROCESS.returncode
#print "HTTPSERVEPROCESS value: %s" % self.HTTPSERVERPROCESS.returncode
except Exception as e:
print "Exception captured while killing HTTP Server process : %s\n" % e

def startCapture(self):
print "********Starting Parallel execution of Server initiation and firefox navigation script*********"
t1 = threading.Thread(target=self.startHTTPServer())
t2 = threading.Thread(target=self.startNavigatingFromBrowser())

Note: Execution starts by calling startCapture()

The problem is that I get following in my terminal upon running the above code

********Starting Parallel execution of HTTP Server initiation and firefox navigation script*********
********HTTP Server started*********
********Opening firefox to start navigation*********

Process finished with exit code 0

My program finishes execution even when the thread started for startNavigatingFromBrowser() is still active. I can see that the firefox is navigating through the website even after I get "Process finished with exit code 0" for my program. Due to this I am not able to detect when my browser navigation thread finishes execution**(This is necessary because I am using process.returncode returned from navigationScript subprocess to kill my HTTP Server process)**.

What changes should I make to the code so that I can successfully detect when the selenium navigation subprocess is finished executing so that I can stop my HTTP Server?

Answer Source

Call t2.join() before exiting your program. This waits for the navigation thread to terminate, before execution continues.


Your navigation thread terminates immediately because you are not waiting for the child process to exit. This should solve the issue:

process=subprocess.Popen('python', \
                        shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

This will halt the thread until the subprocess is done.