Jonathan David Arndt Jonathan David Arndt - 8 days ago 4
Bash Question

Newlines in shell script variable not being replaced properly

Situation: Using a shell script (bash/ksh), there is a message that should be shown in the console log, and subsequently sent via email.

Problem: There are newline characters in the message.

Example below:

ErrMsg="File names must be unique. Please correct and rerun.
Duplicate names are listed below:
File 1.txt
File 1.txt
File 2.txt
File 2.txt
File 2.txt"
echo "${ErrMsg}"
# OK. After showing the message in the console log, send an email


Question: How can these newline characters be translated into HTML line breaks for the email?

Constraint: We must HTML email. Downstream processes (such as Microsoft Outlook) are too inconsistent for anything else to be of use. Simple text email is a good choice, but off the table for this situation.

To be clear, the newlines do not need to be completely removed, but HTML line breaks must be inserted wherever there is a newline character.

This question is being asked because I have already attempted to use several commands, such as
sed
,
tr
, and
awk
with varying degrees of success.

Answer

TL;DR: The following snippet will do the job:

ErrMsg=`echo "$ErrMsg"|awk 1 ORS='<br/>'`

Just make sure there are double quotes around the variable when using echo.


This turned out to be a tricky situation. Some notes of explanation are below.

Using sed

Turns out, sed reads through input line by line, which makes finding and replacing those newlines somewhat outside the norm. There were several clever tricks that appeared to work, but I felt they were far too complicated to apply appropriately to this rather simple situation.

Using tr

According to this answer the tr command should work. Unfortunately, this only translates character by character. The two character strings are not the same length, and I am limited to translating the newline into a space or other single character.

For the following:

ErrMsg="Line 1
Line 2
"
ErrMsg=`echo $ErrMsg| tr '\n' 'BREAK'`

# You might expect:
# "Line 1BREAKLine 2BREAK"
# But instead you get:
# "Line 1BLine 2B"
echo "${ErrMsg}"

Using awk

Using awk according to this answer initially appeared to work, but due to some other circumstances with echo there was a subtle problem. The solution is noted in this forum.

You must have double-quotes around your variable, or echo will strip out all newlines.
(Of course, awk will receive the characters with a newline at the end, because that's what echo does after it echos stuff.)

This snippet is good: (line breaks in the middle are preserved and replaced correctly)

ErrMsg=`echo "$ErrMsg"|awk 1 ORS='<br/>'`

This snipped is bad: (newlines converted to spaces by echo, one line break at end)

ErrMsg=`echo $ErrMsg|awk 1 ORS='<br/>'`