Marco Fedele Marco Fedele - 1 month ago 12
Perl Question

Linux console perl replace not working on large file

On ubuntu I'm running a console perl replace on a csv file of ~500MB. This is the call:

perl -i -pe 's/AS100\n/AS100/g' test.csv


Before run it on the complete file, I extracted a subset of it of ~30MB and run this script successfully.

When running on the full file, no substitution is done, and no error or message is showed.

I've tried also with sed, but the behavior is the same.

How can I solve this issue?

Thank you

Answer

After trying everything, I found out that in the original file the pattern was:

As100\r

and that the \n was a conversion done by Sublime Text when saving the test file.

So the correct code to do the trick was:

perl -i -pe 's/AS100\r/AS100/g' test.csv