zadrozny zadrozny - 1 year ago 65
Python Question

Using Python to kick off Python subprocesses without delay

I want to launch multiple instances of a python script using
, but the kick off script waits for each to complete. How do I prevent it from waiting from going one by one, without waiting for the previous job to complete?

step = 5
for n in range(5, 11, step):["python", cwd + "/" + "", str(n - step), str(n)])

Answer Source

That's the documented behaviour of so you can't use it that way. Instead you can use subprocess.Popen().

import subprocess
import os.path

processes = []
step = 5
for n in range(5, 11, step):
    processes.append(subprocess.Popen(['python', os.path.join(cwd, ''), str(n - step), str(n)]))

for p in processes:    # wait for the child processes to terminate, avoid zombies

Note that it is a bad idea to name a file, especially if it is in the same directory as your main script - an import subprocess will import the local version, not the system version. I've renamed it to in the above code.

It is also important that the parent process waits for the child processes. Omitting this can lead to "zombie" processes in Linux.

If you are using Python 3 you could investigate use of the asyncio module.