Ramon Ramon - 1 year ago 63
Linux Question

Determining time since process's last output - with subprocess.Popen

I am writing a watchdog, of sorts, for processes in a test suite. I need to determine if a test hangs.

I could simply start the process with

, and use
and keep my own timer. However, the tests differ greatly in execution time, which makes it impossible to have a good 'timeout' value that is sensible for all tests.

I have found that a good way to determine if a test has hung is to have a 'timeout' for the last time the process output anything. To that end, I considered using

process = subprocess.Popen(args='<program>', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ...)

, to determine when
are not
. The problem is that
, without a 'timeout' will just wait until the process terminates, and with a 'timeout' will raise a
exception, from which I can't determine if anything was read.
is empty, BTW.

I could not find anything in the documentation that allows one to perform the 'reads' manually. Also, there is usually a lot of output from the process, so starting it with
would be beneficial, as I would have no concern for overflowing pipe buffers.


return a "readable stream object", which one can use to manually poll/select and read. I ended up using select 'Polling Objects', which use the
system call, as bellow:

import os
import select
import subprocess

p = subprocess.Popen(args="<program>", shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
poll_obj = select.poll()
poll_obj.register(p.stdout, select.POLLIN)
poll_obj.register(p.stderr, select.POLLIN)

while p.poll() is None:
events = True
while events:
events = poll_obj.poll(10)
for fd, event in events:
if event & select.POLLIN:
print("STDOUT: " if fd == p.stdout.fileno() else "STDERR: ")
print(os.read(fd, 1024).decode())
# else some other error (see 'Polling Objects')

Answer Source

This is kind of covered here..

Essentially you need to use select() to poll the fd's to see if they have input:

import os                                                                       
import select                                                                   
import subprocess                                                               

p = subprocess.Popen("/bin/sh -c 'while true; do echo hello; sleep 1; >&2 echo world; sleep 1; done'", stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)

while p.poll() == None:                                                         
    fds = select.select([p.stdout, p.stderr], [], [], 100)                      
    print fds                                                                   
    for fd in fds[0]:                                                           
        if fd == p.stdout:                                                      
            print "STDOUT: ",                                                   
        if fd == p.stderr:                                                      
            print "STDERR: ",                                                   
        print os.read(fd.fileno(), 1024),


([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb810>], [], [])
STDOUT:  hello
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb6f0>], [], [])
STDERR:  world
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb810>], [], [])
STDOUT:  hello
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb6f0>], [], [])
STDERR:  world   

os.read() is used instead of fd.read() because you need to read in a non-line oriented way. fd.read() waits until a newline is found -- but then you'll possibly block. With this method you can also split your stderr and stdout.