Nishant Nishant - 6 months ago 30
Bash Question

How do pipes and infinite streams work?

I usually use

command1 | command2 | command3
a lot in Linux but most of them are dealing with finite content.

When I tried this with an infinite stream
cat | sed '' | sed ''
, which hopefully simulates an infinite stream, it didn't work utill I terminated it with
Ctrl-D
. I can solve the problem with using
cat | sed -e '' -e ''
but I would like to know why the first one doesn't work.
cat | cat | cat
works just fine. Is it something to do with
sed
, if so what is that problem?

I tried to think about this problem and the only thing I found different was that when I am using
cat
I hit the Enter key which does something special that is not happening in the first
sed ''
above?

Can anyone let me know how to make pipes work seamlessly with infinite streams?

Answer

Basically a process that reads from a pipe can consume the data byte by byte as soon as they are available in the pipe. However, as long as the programs are using std io functions of the libc, like read, write etc, the libc will buffer the input/output of those programs depending on whether a program is writing to a terminal or not.

By default, if a program is writing to a terminal the libc will buffer the output line wise, if it goes not to a terminal it get's buffered block wise.

You can influence that behaviour using the stdbuf command, like this:

stdbuf -oL cat | stdbuf -ioL sed '' | stdbuf -iL sed ''

I'm using a line based output buffer for the cat command, a line based input and output buffer for the first sed command and a line based input buffer for the last sed command.