alfredodeza alfredodeza - 2 months ago 7
Python Question

subprocess.Popen handling stdout and stderr as they come

I'm trying to process both

stdout
and
stderr
from a
subprocess.Popen
call that captures both via
subprocess.PIPE
but would like to handle the output (for example printing them on the terminal) as it comes.

All the current solutions that I've seen will wait for the completion of the
Popen
call to ensure that all of the
stdout
and
stderr
is captured so that then it can be processed.

This is an example Python script with mixed output that I can't seem to replicate the order when processing it in real time (or as real time as I can):

$ cat mix_out.py

import sys

sys.stdout.write('this is an stdout line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')


The one approach that seems that it might work would be using threads, because then the reading would be asynchronous, and could be processed as
subprocess
is yielding the output.

The current implementation of this just process
stdout
first and
stderr
last, which can be deceiving if the output was originally alternating between both:

cmd = ['python', 'mix_out.py']

process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True,
**kw
)

if process.stdout:
while True:
out = process.stdout.readline()
if out == '' and process.poll() is not None:
break
if out != '':
print 'stdout: %s' % out
sys.stdout.flush()

if process.stderr:
while True:
err = process.stderr.readline()
if err == '' and process.poll() is not None:
break
if err != '':
print 'stderr: %s' % err
sys.stderr.flush()


If I run the above (saved as
out.py
) to handle the
mix_out.py
example script from above, the streams are (as expected) handled in order:

$ python out.py
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line


I understand that some system calls might buffer, and I am OK with that, the one thing I am looking to solve is respecting the order of the streams as they happened.

Is there a way to be able to process both
stdout
and
stderr
as it comes from
subprocess
without having to use threads? (the code gets executed in restricted remote systems where threading is not possible).

The need to differentiate stdout from stderr is a must (as shown in the example output)

Ideally, no extra libraries would be best (e.g. I know
pexpect
solves this)

A lot of examples out there mention the use of
select
but I have failed to come up with something that would preserve the order of the output with it.

Answer

I was able to solve this by using select.select()

process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    close_fds=True,
    **kw
)

while True:
    reads, _, _ = select(
        [process.stdout.fileno(), process.stderr.fileno()],
        [], []
    )

    for descriptor in reads:
        if descriptor == process.stdout.fileno():
            read = process.stdout.readline()
            if read:
                print 'stdout: %s' % read

        if descriptor == process.stderr.fileno():
            read = process.stderr.readline()
            if read:
                print 'stderr: %s' % read
        sys.stdout.flush()

    if process.poll() is not None:
        break

By passing in the file descriptors to select() on the reads argument (first argument for select()) and looping over them (as long as process.poll()indicated that the process was still alive).

No need for threads. Code was adapted from this stackoverflow answer