tiborjan tiborjan - 1 month ago 5
Linux Question

Combined linux commands not writing outfile.txt

I want to search in a log file (real-time) for the contents of msg="yxyxyx" and then write the unique values to outfile.txt

I wrote this:

tail -f /var/log/firewall.log | sed -n '/msg=/{s/.*msg=//;s/\S*=.*//;p}' | awk '!seen[$0]++' >> outfile.txt


Outfile is created but empty. It's writing perfectly to the console without >> outfile.txt part.

Thank you!

Answer

could you use stdbuf -oL or stdbuf -o0 before awk command as below;

tail -f firewall.log | sed -nu '/msg=/{s/.*msg=//;s/\S*=.*//;p}' | stdbuf -oL awk '!seen[$0]++' >> outfile.txt

tail -f firewall.log | sed -nu '/msg=/{s/.msg=//;s/\S=.*//;p}' | stdbuf -o0 awk '!seen[$0]++' >> outfile.txt

man sed;

-u, --unbuffered

load minimal amounts of data from the input files and flush the output buffers more often

man stdbuf;

stdbuf - Run COMMAND, with modified buffering operations for its standard streams.

-o, --output=MODE adjust standard output stream buffering

If MODE is 'L' the corresponding stream will be line buffered. This option is invalid with standard input.

If MODE is '0' the corresponding stream will be unbuffered.

Eg;

first run this;

user@h$ tail -f firewall.log | sed -nu '/msg=/{s/.*msg=//;s/\S*=.*//;p}' | stdbuf -oL awk '!seen[$0]++' >> outfile.txt

run this another terminal; echo msg="1" >> firewall.log, output as below;

user@h$ tail -f firewall.log | sed -nu '/msg=/{s/.*msg=//;s/\S*=.*//;p}' | stdbuf -oL awk '!seen[$0]++' >> outfile.txt

user@host$ cat outfile.txt
1

run again same echo msg="1" >> firewall.log, output is not change;

After run this ; echo msg="2" >> firewall.log, output as below;

user@host$ tail -f firewall.log | sed -nu '/msg=/{s/.*msg=//;s/\S*=.*//;p}' | stdbuf -oL awk '!seen[$0]++' >> outfile.txt

user@host$ cat outfile.txt

1
2
Comments