Programmer Programmer - 5 months ago 11
Bash Question

Command to check the content of a file as a conditional statement

I am looking for a command which will check content of a file and the if the content is found I will not write it again, if its not found then the same will be written.

For example:

cat sample1.txt
test1 1.0.0.0
test2 1.0.0.0
test1 2.0.0.0


Out of these three entries, I want only the first unique entries to be written into the output file. My output file should be

cat sample_op.txt
test1 1.0.0.0
test2 1.0.0.0


My question here is, without using a loop can I get some command to perform this operation?

Answer

Using awk it is simple:

awk '!seen[$1]++' file

test1 1.0.0.0
test2 1.0.0.0

This awk command uses an associative array seen with the key as $1. FOr each row it will check if $1 is available in seen. If yes then line is printed otherwise value of $1 entry is incremented by 1.