Ramon Ramon - 1 year ago 70
Linux Question

Determining time since process's last output - with subprocess.Popen

I am writing a watchdog, of sorts, for processes in a test suite. I need to determine if a test hangs.

I could simply start the process with

subprocess.Popen(...)
, and use
Popen.wait(timeout=to)
or
Popen.poll()
and keep my own timer. However, the tests differ greatly in execution time, which makes it impossible to have a good 'timeout' value that is sensible for all tests.

I have found that a good way to determine if a test has hung is to have a 'timeout' for the last time the process output anything. To that end, I considered using

process = subprocess.Popen(args='<program>', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ...)


and
Popen.communicate()
, to determine when
stdout
and/or
stderr
are not
None
. The problem is that
Popen.communicate()
, without a 'timeout' will just wait until the process terminates, and with a 'timeout' will raise a
TimeoutExpired
exception, from which I can't determine if anything was read.
TimeoutExpired.output
is empty, BTW.

I could not find anything in the documentation that allows one to perform the 'reads' manually. Also, there is usually a lot of output from the process, so starting it with
stdout=<open_file_descriptor>
would be beneficial, as I would have no concern for overflowing pipe buffers.

Update/Solution:

Popen.stdout
and
Popen.stderr
return a "readable stream object", which one can use to manually poll/select and read. I ended up using select 'Polling Objects', which use the
poll()
system call, as bellow:

import os
import select
import subprocess

p = subprocess.Popen(args="<program>", shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
poll_obj = select.poll()
poll_obj.register(p.stdout, select.POLLIN)
poll_obj.register(p.stderr, select.POLLIN)

while p.poll() is None:
events = True
while events:
events = poll_obj.poll(10)
for fd, event in events:
if event & select.POLLIN:
print("STDOUT: " if fd == p.stdout.fileno() else "STDERR: ")
print(os.read(fd, 1024).decode())
# else some other error (see 'Polling Objects')

Answer Source

This is kind of covered here..

Essentially you need to use select() to poll the fd's to see if they have input:

#!/usr/bin/python                                                               
import os                                                                       
import select                                                                   
import subprocess                                                               

p = subprocess.Popen("/bin/sh -c 'while true; do echo hello; sleep 1; >&2 echo world; sleep 1; done'", stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)

while p.poll() == None:                                                         
    fds = select.select([p.stdout, p.stderr], [], [], 100)                      
    print fds                                                                   
    for fd in fds[0]:                                                           
        if fd == p.stdout:                                                      
            print "STDOUT: ",                                                   
        if fd == p.stderr:                                                      
            print "STDERR: ",                                                   
        print os.read(fd.fileno(), 1024),

Output:

([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb810>], [], [])
STDOUT:  hello
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb6f0>], [], [])
STDERR:  world
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb810>], [], [])
STDOUT:  hello
([<open file '<fdopen>', mode 'rb' at 0x7f23db5eb6f0>], [], [])
STDERR:  world   

os.read() is used instead of fd.read() because you need to read in a non-line oriented way. fd.read() waits until a newline is found -- but then you'll possibly block. With this method you can also split your stderr and stdout.