philipper philipper - 4 months ago 10
Bash Question

shell multipipe broken with multiple python scripts

I am trying to get the stdout of a python script to be shell-piped in as stdin to another python script like so:

find ~/test -name "*.txt" | python my_multitail.py | python line_parser.py


It should print an output but nothing comes out of it.

Please note that this works:

find ~/test -name "*.txt" | python my_multitail.py | cat


And this works too:

echo "bla" | python line_parser.py


my_multitail.py prints out the new content of the .txt files:

from multitail import multitail
import sys

filenames = sys.stdin.readlines()
# we get rid of the trailing '\n'
for index, filename in enumerate(filenames):
filenames[index] = filename.rstrip('\n')

for fn, line in multitail(filenames):
print '%s: %s' % (fn, line),
sys.stdout.flush()


When a new line is added to the .txt file ("hehe") then my_multitail.py prints:


/home/me/test2.txt: hehe


line_parser.py simply prints out what it gets on stdin:

import sys

for line in sys.stdin:
print "line=", line


There is something I must be missing. Please community help me :)

Answer

There's a hint if you run your line_parser.py interactively:

$ python line_parser.py 
a
b
c
line= a

line= b

line= c

Note that I hit ctrl+D to provoke an EOF after entering the 'c'. You can see that it's slurping up all the input before it starts iterating over the lines. Since this is a pipeline and you're continuously sending output through to it, this doesn't happen and it never starts processing. You'll need to choose a different way of iterating over stdin, for example:

import sys

line = sys.stdin.readline()
while line:
    print "line=", line
    line = sys.stdin.readline()
Comments