Merging CSV files : Appending instead of merging
Thank you so much @wahwahwah. I used your script to make nautilus-action, but it work correctly only with this changes:
#!/bin/bash
for last; do true; done
OutFileName=$last/RESULT_`date +"%d-%m-%Y"`.csv # Fix the output name
i=0 # Reset a counter
for filename in "$last/"*".csv"; do
if [ "$filename" != "$OutFileName" ] ; # Avoid recursion
then
if [[ $i -eq 0 ]] ; then
head -1 "$filename" > "$OutFileName" # Copy header if it is the first file
fi
tail -n +2 "$filename" >> "$OutFileName" # Append from the 2nd line each file
i=$(( $i + 1 )) # Increase the counter
fi
done
Alternative simple answer, this as combine_csv.sh:
#!/bin/bash
{ head -n 1 $1 && tail -q -n +2 $*; }
can be used like this:
pattern="my*filenames*.csv"
combine_csv.sh ${pattern} > result.csv
Assuming that all the csv files have the same format and all start with the same header, you can write a little script as the following to append all files in only one and to take only one time the header.
#!/bin/bash
OutFileName="X.csv" # Fix the output name
i=0 # Reset a counter
for filename in ./*.csv; do
if [ "$filename" != "$OutFileName" ] ; # Avoid recursion
then
if [[ $i -eq 0 ]] ; then
head -1 "$filename" > "$OutFileName" # Copy header if it is the first file
fi
tail -n +2 "$filename" >> "$OutFileName" # Append from the 2nd line each file
i=$(( $i + 1 )) # Increase the counter
fi
done
Notes:
- The
head -1
orhead -n 1
command print the first line of a file (the head). - The
tail -n +2
prints the tail of a file starting from the lines number 2 (+2
) - Test
[ ... ]
is used to exclude the output file from the input list. - The output file is rewritten each time.
- The command
cat a.csv b.csv > X.csv
can be simply used to append a.csv and b csv in a single file (but you copy 2 times the header).
The paste
command pastes the files one on a side of the other. If a file has white spaces as lines you can obtain the output that you reported above.
The use of -d ,
asks to paste command
to define fields separated by a comma ,
, but this is not the case for the format of the files you reported above.
The cat
command instead concatenates files and prints on the standard output, that means it writes one file after the other.
Refer to man head
or man tail
for the syntax of the single options (some version allows head -1
other instead head -n 1
)...