Yuka Takemon Yuka Takemon - 3 months ago 14
Linux Question

While loop cp give copies of partial file (linux)

I am trying to copy a list of file in varying directories based on their sample name using the following script. Although the files are copied, the file are only partially copied. I have 64k lines in each file, but only exactly 40k lines are copied.

while read sample
do
echo copying ${sample}
cp ${sample_dir}/*${sample}*/file.tsv ${output_dir}/${sample}.file.tsv
done < ${input_list}/sample_list.txt


Am I missing something obvious here? Does the cp command have limits on how many lines it can copy?

Cheers,

Answer

I don't think CP command has limit on file size (unless ulimit has restrictions) to copy but it has limit on number of files to copy or check if it is related to new file creation with larger size.

check limits across the system using command ulimit,

ulimit -a

And verify the file size is not limited to 40k and if it is unlimited then no issue (like, file size (blocks, -f) unlimited).

Try with rsync command if it works for you,

rsync -avh source destination

Could you verify few things,

1) Verify if it's not file read error, like cat the file and save the output to another file (either manually/redirect).

cat *.tsv > /tmp/verify-size

And then verify the size on that file,

du -h /tmp/verify-size ---> This should be 64k

2) Create large dummy file with size > 40k (or exact size of .tsv (64k))

dd if=/dev/zero of=/tmp/verify-newfile-size bs=64000 count=1

du -h /tmp/verify-newfile-size  ---> This should be 64k

If this new file creation success (if you are able to create a file with any size) then try CP command and verify the size

-OR- Try with dd command

dd if=/tmp/verify-newfile-size of=/tmp/verify-newfile-size2 bs=64000 count=1

3) Try with an atomic command,

mv /tmp/verify-newfile-size /tmp/verify-newfile-size3
Comments