johnathon doeth johnathon doeth - 1 year ago 46
Bash Question

Bash - Piping output of command into while loop

I'm writing a Bash script where I need to look through the output of a command and do certain actions based on that output. For clarity, this command will output a few million lines of text and it may take roughly an hour or so to do so.

Currently, I'm executing the command and piping it into a while loop that reads a line at a time then looks for certain criteria. If that criterion exists, then update a .dat file and reprint the screen. Below is a snippet of the script.

eval "$command"| while read line ; do
if grep -Fq "Specific :: Criterion"; then
#pull the sixth word from the line which will have the data I need
temp=$(echo "$line" | awk '{ printf $6 }')
#sanity check the data
echo "\$line = $line"
echo "\$temp = $temp"

#then push $temp through a case statement that does what I need it to do.
fi
done


So here's the problem, the sanity check on the data is showing weird results. It is printing lines that don't contain the grep criteria.

To make sure that my grep statement is working properly, I grep the log file that contains a record of the text that is output by the command and it outputs only the lines that contain the specified criteria.

I'm still fairly new to Bash so I'm not sure what's going on. Could it be that the command is force feeding the while loop a new $line before it can process the $line that met the grep criteria?

Any ideas would be much appreciated!

Answer Source

How does grep know what line looks like?

if ( echo $line | grep -Fq "Specific :: Criterion"); then

But I cant help feel like you are overcomplicating a lot.

function process() { 
    echo "I can do anything I want"
    echo " per element $1"
    echo " that I want here"
}

export -f process

$command | grep -F "Specific :: Criterion" | awk '{print $6}' | xargs -I % -n 1 bash -c "process %";

Run the command, filter only matching lines, and pull the sixth element. Then if you need to run an arbitrary code on it, send it to a function (you export to make it visible in subprocesses) via xargs.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download