zadrozny zadrozny - 3 months ago 11
Python Question

Using Python to kick off Python subprocesses without delay

I want to launch multiple instances of a python script using

subprocess.call
, but the kick off script waits for each to complete. How do I prevent it from waiting from going one by one, without waiting for the previous job to complete?

step = 5
for n in range(5, 11, step):
subprocess.call(["python", cwd + "/" + "subprocess.py", str(n - step), str(n)])

Answer

That's the documented behaviour of subprocess.call() so you can't use it that way. Instead you can use subprocess.Popen().

import subprocess
import os.path

processes = []
step = 5
for n in range(5, 11, step):
    processes.append(subprocess.Popen(['python', os.path.join(cwd, 'child.py'), str(n - step), str(n)]))

for p in processes:    # wait for the child processes to terminate, avoid zombies
    p.wait()

Note that it is a bad idea to name a file subprocess.py, especially if it is in the same directory as your main script - an import subprocess will import the local version, not the system version. I've renamed it to child.py in the above code.

It is also important that the parent process waits for the child processes. Omitting this can lead to "zombie" processes in Linux.

If you are using Python 3 you could investigate use of the asyncio module.

Comments