Nishant Nishant - 1 year ago 78
Bash Question

How do pipes and infinite streams work?

I usually use

command1 | command2 | command3
a lot in Linux but most of them are dealing with finite content.

When I tried this with an infinite stream
cat | sed '' | sed ''
, which hopefully simulates an infinite stream, it didn't work utill I terminated it with
. I can solve the problem with using
cat | sed -e '' -e ''
but I would like to know why the first one doesn't work.
cat | cat | cat
works just fine. Is it something to do with
, if so what is that problem?

I tried to think about this problem and the only thing I found different was that when I am using
I hit the Enter key which does something special that is not happening in the first
sed ''

Can anyone let me know how to make pipes work seamlessly with infinite streams?

Answer Source

Basically a process that reads from a pipe can consume the data byte by byte as soon as they are available in the pipe. However, as long as the programs are using std io functions of the libc, like read, write etc, the libc will buffer the input/output of those programs depending on whether a program is writing to a terminal or not.

By default, if a program is writing to a terminal the libc will buffer the output line wise, if it goes not to a terminal it get's buffered block wise.

You can influence that behaviour using the stdbuf command, like this:

stdbuf -oL cat | stdbuf -ioL sed '' | stdbuf -iL sed ''

I'm using a line based output buffer for the cat command, a line based input and output buffer for the first sed command and a line based input buffer for the last sed command.