Surya Surya - 2 days ago 5
Bash Question

removing duplicate lines from file /grep

I want to remove all lines where all the second column 05408736032 are same

0009300|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|asdf| 0009367|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|adff|

these lines are not consecutive. Its fine to remove all the lines . I dont have to keep one of them around.

Sorry my unix fu is really weak from non usage :) .

Answer

If the columns are not fixed width, you can still use sort:

sort -t '|' --key=10,10 -g FILENAME
  1. The -t flag will set the separator.
  2. The -g is just for natural numeric ordering.
Comments