DonPromillo DonPromillo - 1 year ago 36
Linux Question

updating a file using tee randomly fails in linux bash script

when using

sed -e
to update some parameters of a config file and pipe it to
| tee
(to write the updated content into the file), this randomly breaks and causes the file to be invalid (size 0).

In Summary, this code is used for updating parameters:

# based on the provided linenumber, add some comments, add the new value, delete old line

sed -e "$lineNr a # comments" -e "$lineNr a $newValue" -e "$lineNr d" $myFile | sudo tee $myFile

I set up an script which calls this update command 100 times.

  • In a Ubuntu VM (Parallels Desktop) on a shared Directory with OSX this
    behaviour occurs up to 50 times

  • In a Ubuntu VM (Parallels Desktop) on the
    Ubuntu partition this behaviour occurs up to 40 times

  • On a native System (IntelNUC with Ubuntu) this behaviour occurs up to 15 times

Can someone explain why this is happening?

Here is a fully functional script where you can run the experiment as well. (All necessary files are generated by the script, so you can simply copy/paste it into a bashscriptfile and run it)

# main function at bottom


# This method updates parameters with a new value. The replacement is performed linewise.
local valueOfInterest="$1"
local newValue="$2"
local filePath="$3"

# stores all matching linenumbers
local listOfLines=""
# stores the linenumber which is going to be replaced
local lineToReplace=""

# find value of interest in all non-commented lines and store related lineNumber
lineToReplace=$( grep -nr "^[^#]*$valueOfInterest" $filePath | sed -n 's/^\([0-9]*\)[:].*/\1/p' )

# Update parameters
# replace the matching line with the desired value
oldValue=$( sed -n "$lineToReplace p" $filePath )
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" $filePath | sudo tee $filePath >/dev/null

# Sanity check to make sure file did not get corrupted by updating parameters
if [[ ! -s $filePath ]] ; then
echo "[ERROR]: While updating file it turned invalid."
return 31


#=== Actual Update Function ====

echo -n "Update Parameter1 ..."
doUpdateParameterInFile "Parameter1" "Parameter1 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 33 ; fi

echo -n "Update Parameter2 ..."
doUpdateParameterInFile "Parameter2" "Parameter2=90" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 34 ; fi

echo -n "Update Parameter3 ..."
doUpdateParameterInFile "Parameter3" "Parameter3 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 35 ; fi

#=== Main Loop ===

#generate file config.txt
printf "# Configfile with 3 Parameters\n#[Parameter1]\n#only takes YES or NO\nParameter1 NO \n\n#[Parameter2]\n#Parameter2 takes numbers\nParameter2 = 100 \n\n#[Parameter3]\n#Parameter3 takes YES or NO \nParameter3 YES\n" > config.txt
cp config.txt config.txt.bkup

# Start the experiment and let it run 100 times
while [[ $cnt != "100" ]] ; do
echo "==========run: $cnt; fails: $failSum======="
if [[ $? != "0" ]] ; then cp config.txt.bkup config.txt ; failSum=$(($failSum+1)) ; fi
sleep 0.5


Answer Source

The problem is that you're using tee to overwrite $filepath at the same time as sed is trying to read from it. If tee truncates it first then sed gets an empty file and you end up with a 0 length file at the other end.

If you have GNU sed you can use the -i flag to have sed modify the file in place (other versions support -i but require an argument to it). If your sed doesn't support it you can have it write to a temp file and move it back to the original name like

sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" "$filePath" > "$tmpname"
sudo mv "$tmpname" "$filePath"

or if you want to preserve the original permissions you could do

sudo sh -c "cat '$tmpname' > '$filePath'"
rm "$tmpname"

or use your tee approach like

sudo tee "$filePath" >/dev/null <"$tmpname"
rm "$tmpname"