RandomUser RandomUser - 6 months ago 23
Linux Question

exclude columns from awk

I am trying to delete few columns and then to unique of the file contents. Columns which I want to delete are like month,day,time and epoch time;these are different in each line and cannot let me to unique of the file contents.

Sample contents of sample.log :

Jun 5 05:13:13 AAA AAA AAAA 1433495593.306611 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
Jun 5 05:13:14 AAA AAA AAAA 1433495594.306612 XXXX CCCC CCCC AAAA SDDDD DFFFFF222
Jun 5 05:13:13 AAA AAA AAAA 1433495593.306611 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
Jun 5 05:13:15 AAA AAA AAAA XXXXX 1433495596.306614 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
Jun 5 05:13:16 AAA AAA AAAA XXXXX 1433495597.306615 XXXX CCCC CCCC AAAA SDDDD DFFFFF333
Jun 5 05:13:17 AAA AAA AAAA XXXXX 1433495598.306616 XXXX CCCC CCCC AAAA SDDDD DFFFFF444


Issue:

Month, date,time are in fixed column , however epoch time is toggling between column number 7 and 8. Want to know how to deal with this.

Sample output:

Jun 5 05:13:13 AAA AAA AAAA 1433495593.306611 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
Jun 5 05:13:13 AAA AAA AAAA 1433495593.306611 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
Jun 5 05:13:15 AAA AAA AAAA XXXXX 1433495596.306614 XXXX CCCC CCCC AAAA SDDDD DFFFFF111


If above is too much to ask then like below:

AAA AAA AAAA 1433495593.306611 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
AAA AAA AAAA 1433495593.306611 XXXX CCCC CCCC AAAA SDDDD DFFFFF111
AAA AAA AAAA XXXXX 1433495596.306614 XXXX CCCC CCCC AAAA SDDDD DFFFFF111


I am trying things in following direction but not very helpful.

while read line
do

seven=$(echo $line |awk '{print $7}')
eight=$(echo $line |awk '{print $8}')

if [[ "$seven" =~ "^[0-9]" ]];then
#echo "seventh column starts with number"
echo $line|awk '$1=$2=$3=$7=" " {print}'
else
#echo "Eighth column starts with number"
echo $line|awk '$1=$2=$3=$8=" " {print}'
fi
done < $1


More example:

Input file contents:

Jun 5 05:13:13 AAA BBB CCC 142222222222.000 DDD EEE FFFF
Jun 5 05:13:13 AAA BBB CCC 142222222223.000 DDD EEE FFFF
Jun 5 05:13:14 AAA BBB CCC 142222222224.000 DDD EEE GGGG
Jun 5 05:13:13 AAA BBB CCC XXX 142222222225.000 DDD EEE GGGG
Jun 5 05:13:13 AAA BBB CCC XXX 142222222225.000 DDD EEE FFFF
Jun 5 05:13:13 AAA BBB CCC XXX 142222222226.000 DDD EEE FFFF


Output:

Jun 5 05:13:13 AAA BBB CCC 142222222223.000 DDD EEE FFFF
Jun 5 05:13:13 AAA BBB CCC 142222222223.000 DDD EEE GGGG
Jun 5 05:13:13 AAA BBB CCC XXX 142222222225.000 DDD EEE GGGG
Jun 5 05:13:13 AAA BBB CCC XXX 142222222225.000 DDD EEE FFFF


OR

Output:

AAA BBB CCC DDD EEE FFFF
AAA BBB CCC DDD EEE GGGG
AAA BBB CCC XXX DDD EEE GGGG
AAA BBB CCC XXX DDD EEE FFFF

Answer

If I'm understanding the question correctly there is no need for Bash here, just Awk:

% awk '
{
    for (f = 4; f <= NF; ++f) { # Start at column 4
        if (f == 7 || f == 8) { # Treat columns 7 or 8 differently
            if ($f !~ /^[0-9]+\.[0-9]+$/) { # Only print if non-numeric
                printf $f " "
            }
        } else {
            printf $f " "
        }
    }
    printf "\n"
}
' sample.log          
AAA AAA AAAA XXXX CCCC CCCC AAAA SDDDD DFFFFF111 
AAA AAA AAAA XXXX CCCC CCCC AAAA SDDDD DFFFFF222 
AAA AAA AAAA XXXX CCCC CCCC AAAA SDDDD DFFFFF111 
AAA AAA AAAA XXXXX XXXX CCCC CCCC AAAA SDDDD DFFFFF111 
AAA AAA AAAA XXXXX XXXX CCCC CCCC AAAA SDDDD DFFFFF333 
AAA AAA AAAA XXXXX XXXX CCCC CCCC AAAA SDDDD DFFFFF444 

To grab the unique rows:

% awk '             
{
    for (f = 4; f <= NF; ++f) { # Start at column 4
        if (f == 7 || f == 8) { # Treat columns 7 or 8 differently
            if ($f !~ /^[0-9]+\.[0-9]+$/) { # Only print if non-numeric
                printf $f " "
            }
        } else {
            printf $f " "
        }
    }
    printf "\n"
}
' sample2.log | sort -u
AAA BBB CCC DDD EEE FFFF 
AAA BBB CCC DDD EEE GGGG 
AAA BBB CCC XXX DDD EEE FFFF 
AAA BBB CCC XXX DDD EEE GGGG 
Comments