So basically i want to merge a couple of CSV files. Im using the following script to do that :
paste -d , *.csv > final.txt
Mon Jul 07 20:43:47 +0000 2014 4.86249E+17
Mon Jul 07 19:58:29 +0000 2014 4.86238E+17
Mon Jul 07 19:42:33 +0000 2014 4.86234E+17
CreatedAt ID CreatedAt ID
Mon Jul 07 20:43:47 +0000 2014 4.86249E+17 Mon Jul 07 18:25:53 +0000 2014 4.86215E+17
Mon Jul 07 19:58:29 +0000 2014 4.86238E+17 Mon Jul 07 17:19:18 +0000 2014 4.86198E+17
Mon Jul 07 19:42:33 +0000 2014 4.86234E+17 Mon Jul 07 15:45:13 +0000 2014 4.86174E+17
Mon Jul 07 15:34:13 +0000 2014 4.86176E+17
Assuming that all the csv files have the same format and all start with the same header, you can write a little script as the following to append all files in only one and to take only one time the header.
#!/bin/bash OutFileName="X.csv" # Fix the output name i=0 # Reset a counter for filename in ./*.csv; do if [ "$filename" != "$OutFileName" ] ; # Avoid recursion then if [[ $i -eq 0 ]] ; then head -1 $filename > $OutFileName # Copy header if it is the first file fi tail -n +2 $filename >> $OutFileName # Append from the 2nd line each file i=$(( $i + 1 )) # Increase the counter fi done
head -n 1command print the first line of a file (the head).
tail -n +2prints the tail of a file starting from the lines number 2 (
[ ... ]is used to exclude the output file from the input list.
cat a.csv b.csv > X.csvcan be simply used to append a.csv and b csv in a single file (but you copy 2 times the header).
paste command pastes the files one on a side of the other. If a file has white spaces as lines you can obtain the output that you reported above.
The use of
-d , asks to
paste command to define fields separated by a comma
,, but this is not the case for the format of the files you reported above.
cat command instead concatenates files and prints on the standard output, that means it writes one file after the other.
man head or
man tail for the syntax of the single options (some version allows
head -1 other instead
head -n 1)...