Ramon Ramon - 5 months ago 26
Linux Question

Determining time since process's last output - with subprocess.Popen

I am writing a watchdog, of sorts, for processes in a test suite. I need to determine if a test hangs.

I could simply start the process with

subprocess.Popen(...)
, and use
Popen.wait(timeout=to)
or
Popen.poll()
and keep my own timer. However, the tests differ greatly in execution time, which makes it impossible to have a good 'timeout' value that is sensible for all tests.

I have found that a good way to determine if a test has hung is to have a 'timeout' for the last time the process output anything. To that end, I considered using

process = subprocess.Popen(args='<program>', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ...)


and
Popen.communicate()
, to determine when
stdout
and/or
stderr
are not
None
. The problem is that
Popen.communicate()
, without a 'timeout' will just wait until the process terminates, and with a 'timeout' will raise a
TimeoutExpired
exception, from which I can't determine if anything was read.
TimeoutExpired.output
is empty, BTW.

I could not find anything in the documentation that allows one to perform the 'reads' manually. Also, there is usually a lot of output from the process, so starting it with
stdout=<open_file_descriptor>
would be beneficial, as I would have no concern for overflowing pipe buffers.

Update/Solution:

Popen.stdout
and
Popen.stderr
return a "readable stream object", which one can use to manually poll/select and read. I ended up using select 'Polling Objects', which use the
poll()
system call, as bellow:

import os
import select
import subprocess

p = subprocess.Popen(args="<program>", shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
poll_obj = select.poll()
poll_obj.register(p.stdout, select.POLLIN)
poll_obj.register(p.stderr, select.POLLIN)

while p.poll() is None:
events = True
while events:
events = poll_obj.poll(10)
for fd, event in events:
if event & select.POLLIN:
print("STDOUT: " if fd == p.stdout.fileno() else "STDERR: ")
print(os.read(fd, 1024).decode())
# else some other error (see 'Polling Objects')

Answer

This is kind of covered here..

Essentially you need to use select() to poll the fd's to see if they have input:

#!/usr/bin/python                                                               
import os                                                                       
import select                                                                   
import subprocess                                                               

p = subprocess.Popen("/bin/sh -c 'while true; do echo hello; sleep 1; >&2 echo world; sleep 1; done'", stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)

while p.poll() == None:                                                         
    fds = select.select([p.stdout, p.stderr], [], [], 100)                      
    print fds                                                                   
    for fd in fds[0]:                                                           
        if fd == p.stdout:                                                      
            print "STDOUT: ",                                                   
        if fd == p.stderr:                                                      
            print "STDERR: ",                                                   
        print os.read(fd.fileno(), 1024),

Output:

([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb810>], [], [])
STDOUT:  hello
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb6f0>], [], [])
STDERR:  world
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb810>], [], [])
STDOUT:  hello
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb6f0>], [], [])
STDERR:  world   

os.read() is used instead of fd.read() because you need to read in a non-line oriented way. fd.read() waits until a newline is found -- but then you'll possibly block. With this method you can also split your stderr and stdout.