galactica galactica - 2 months ago 10
Linux Question

how to batch sed files and redirect results

I wonder if there is a one-liner for batch processing a set of files in one folder and redirect the results into a different folder.
I tried something like this:

find input_dir/ -name "PATTERN" | xargs -I {} sed 's:foo:bar:g' > output_dir/{}


For example, input_dir/ has file A, B, C and my hoped result is to have processed files A, B, C in output_dir/, with the same file names.

my hope was to use {} to replace file names and to build the output file paths, but this didn't work.

Anyone knows how to fix this? Or other better ways of doing so?
Thanks!

Answer

My technique for this is to write a shell script that does the job, and then run it via find. For example, your actions could be written into a script munger.sh:

#!/bin/sh
for file in "$@"
do
    output="output_dir/$(basename "$file")"
    sed -e 's:foo:bar:g' "$file" > "$output"
done

The find command becomes:

find input_dir -name "PATTERN" -exec sh munger.sh {} +

This runs the script with the file names as arguments, bundling conveniently large number of file names into a single invocation of the shell script. If you're not going to need it again, you can simply remove munger.sh when you're done.

Yes, you can do all sorts of contortions to execute the command the way you want (perhaps using find … -exec bash -c "the script to be executed" arg0 {} +) but it is often harder than writing a relatively simple script and using it and throwing it away. There tend to be fewer problems with quoting, for example, when you run an explicit script than when you try to write the script on the command line. If you find yourself fighting with single quotes, double quotes and backslashes (or back-quotes), then it is time to use a simple script as shown.