emetib emetib - 1 year ago 47
Python Question

Incomplete reads from file written by Popen()-based subprocess

i'm trying to read the output from one function into another one.

if i break things down into two steps, call the first function(journal.py) from the command line, and then call the second(ip_list.py) i get the results that i'm looking for.

if i try to import the first and run it in the second the resulting list is empty.

import re
import journal

ip_list = []
with open('Invalid_names_file') as file1:
a = [re.search(r'((\d+\.)+\d+)', line).group() for line in file1]
for x in a:
if x not in ip_list:


output ->

<_io.TextIOWrapper name='Invalid_names_file' mode='r' encoding='UTF-8'>

when called in this manner, the file that i'm opening is there (once the script is done running), with what i'm expecting, yet i can't get it to read from it when i try to include it with the import.

i have the print()s so that i can try to understand what's happening, yet can't get my head around it.

journal.py is a Popen command that writes the file.

edit for chris


from subprocess import Popen
import os

def journal():
with open('Invalid_names_file', 'w') as Invalid_names_file:
Popen('journalctl -u sshd.service --no-pager --since -168hours\
--until today | grep Invalid', stdout=Invalid_names_file,\
universal_newlines=True, bufsize=1, shell=True)
if os.stat('Invalid_names_file').st_size == 0:
Popen('journalctl -u ssh.service --no-pager --since -168hours\
--until today | grep Invalid', stdout=Invalid_names_file,\
universal_newlines=True, bufsize=1, shell=True)

Answer Source

You should wait for Popen() to finish. Assign its return value to a variable and call wait() on it:

p = Popen('journalctl ...')

When you run the journal script separately, the parent process will only return when all of its children have terminated. However, Popen() doesn't wait – unless you tell it to. Thus, in your case, the journal() function exits immediately after starting the subprocess, so by the time you're reading the target file, it is still empty or incomplete.