gunnersfan gunnersfan - 1 year ago 225
Bash Question

Read and Replace key-value properties using Shell Script

I wanted to replace the keys in my template file (found recursively using find) with the corresponding values. The keys will be loaded from a properties file. See my script below:

Script file Contents:

. /home/uname/config/
echo ${name_1}
echo ${age_1}
echo ${job_1}
for file in $(find /home/uname/config -name "*.txt"); do
echo $file
cp -f $file $temp_file
rm $file
envsubst < $temp_file > $file
done Contents:


template.txt Contents:

My name is ${name_1}, my age is ${age_1} and my job is in ${job_1}.

I am able to load the key/value pairs (the echo statement displays the key values), but when envsubst writes out the template file, it is not replacing any of the keys with the values. The output I get is:

My name is and my age is and my job is in .

Answer Source

envsubst, as the name implies, requires that its key/value pairs be in environment variables. Mere assignments create shell variables, not environment variables.

The following is an attempt at a best-practices replacement:

set -a # turn on auto-export
set -a # turn off auto-export

while IFS= read -r -d '' filename; do
  tempfile=$(mktemp "$filename.XXXXXX")
  if envsubst <"$filename" >"$tempfile"; then
    mv "$tempfile" "$filename"
    rm -f "$tempfile"
done < <(find /home/uname/config -name '*.txt' -print0)

Key points:

  • Quotes are important. Try putting spaces in your filenames and seeing how the original behaves if you're not so sure.
  • Using set -a before sourcing the file ensures that the variables it sets are present in the environment.
  • Using IFS= read -r -d '' filename to iterate over names -- unlike for filename in $(find ...) -- ensures that your code correctly handles filenames with spaces, filenames with glob characters in their names, and other unusual cases.
  • Using mktemp to generate temporary filenames prevents symlink attacks -- think of what would happen if someone else with permissions to write to /home/uname/, but without permissions to write to /etc, created a symlink to /etc/passwd (or any other file you care about) named /home/uname/temp_file.txt. Moreover, using mktemp to generate random names means that multiple concurrent instances of the same script won't stomp on the same temporary filename.
  • Writing the output of envsubst to a temporary file, and then renaming that temporary file to your destination name, ensures that you don't overwrite your output until it's finished being generated.