cqdjyy01234 cqdjyy01234 - 7 months ago 138
Python Question

Real time read from subprocess.stdout on Windows

To emphasize, the problem is real time read instead of non-blocking read. It has been asked before, e.g. subprocess.Popen.stdout - reading stdout in real-time (again). But no satisfactory solution has been proposed.

As an example, the following code tries to simulate the python shell.

import subprocess

p = subprocess.Popen(['python'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

while True:
line = input('>>> ')
p.stdin.write(line.encode())
print('>>> ', p.stdout.read().decode())


However, it would be blocked when reading from
p.stdout
. After searching around, I found the following two possible soutions.


  1. using
    fctrl
    and
    O_NONBLOCK

  2. using
    thread
    and
    queue



Whereas the 1st soution may work and only work on linux, the 2nd soution just turn blocking read to non-blocking read, i.e. I cannot get real time output of the subprocess. For example, if I input '
print("hello")
', I will get nothing from
p.stdout
using 2nd solution.

Perhaps, someone would suggest
p.communite
. Unfortunately, it is not suitable in this case, since it would close stdin as described here.

So, is there any solutions for Windows?

Edited: Even if
-u
is turned on and
p.stdout.read
is replaced with
p.stdout.readline
, the problem still exists.

import subprocess

p = subprocess.Popen(['python', '-u'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

while True:
line = input('>>> ')
p.stdin.write(line.encode())
p.stdin.flush()
print('>>> ', p.stdout.readline().decode())


Solution: The following is the final code based on J.F. Sebastian's answer and comments.

from subprocess import Popen, PIPE, STDOUT

with Popen(
['python', '-i', '-q'],
stdin=PIPE, stdout=PIPE, stderr=STDOUT,
bufsize=0
) as process:
while True:
line = input('>>> ')
if not line:
break
process.stdin.write((line+'\n').encode())
print(process.stdout.readline().decode(), end='')


It should be noted that the program would hang when the command triggers no output.

Answer

Here's a complete working example that uses a subprocess interactively:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE, DEVNULL

with Popen([sys.executable, '-i'], stdin=PIPE, stdout=PIPE, stderr=DEVNULL,
           universal_newlines=True) as process:
    for i in range(10):
        print("{}**2".format(i), file=process.stdin, flush=True)
        square = process.stdout.readline()
        print(square, end='')

Here's another example: how to run [sys.executable, '-u', 'test.py'] interactively.