davis davis - 3 months ago 12x
Bash Question

Single quotation marks in variable call not allowing search in multiple paths

I'm trying to filter some logs and for this I'm using the command below.

eval metrics=($(ssh -q XXXXXX@$server grep -H '' $path | sed 's/-\|log:/|/g' | sed 's/CAM01CPD|\|CAM02BPD|\|XXXX|\|//g' | sed 's;/;|;g' | awk -F\| '{ printf "%s|%s|%s|%s|%s|%s|%s|%s|", $11, $17, $18, $19, $20, $21, $22, $23; if ($26 != "") printf "%s\n", $26; else printf "%s\n", $25;}'))

And the $path in the command above is this:


The problem is that as we have multiples paths I have to use the * in the path to filter all of the directories, but in the script this * is being interpreted as part of the path because of those single quotation marks.

Output from bash -x:

++ ssh -q xxxxxx@xxxxxx grep -H '' '/logs/xxx/production/.snapshot/ifs-QBA_LS1-VAULT_30Day_2016-06-29_19-01/jboss/pvmk*/xxx-production-xxx*/AppLog/xxx-production-xxx*-apimetrics.log'
++ sed 's/CAM01CPD|\|CAM02BPD|\|XXXX|\|//g'
++ sed 's;/;|;g'
++ awk '-F|' '{ printf "%s|%s|%s|%s|%s|%s|%s|%s|", $11, $17, $18, $19, $20, $21, $22, $23; if ($26 != "") printf "%s\n", $26; else printf "%s\n", $25;}'
++ sed 's/-\|log:/|/g'
+ eval 'metrics=()'
++ metrics=()

How do I run this command inside the Shellscript without those quotation marks so I can explore all the paths?


See below for the robust way to pass globbing results to an external command, using an array.

The likely cause of your problem is that your glob doesn't match anything, and is therefore left as-is.

To verify, use your pattern unquoted on the command line and see what you get:

echo /logs/XXX/production/.snapshot/ifs-QBA_LS1-VAULT_30Day_2016-06-29_19-01/jboss/pvmk*/xxx-production-xxx*/AppLog/xxx-production-xxx*-apimetrics.log

If you get the same string - with the * chars. - back, you know that nothing matched.

If you run shopt -s nullglob beforehand, the outcome will be more obvious: if nothing matches, nothing will be echoed.

Using shopt -s nullglob or even shopt -s failglob in your script will allow you to detect up front if you have any matching files.

The single quotes in the output are an artifact of set -x, and they are unrelated to what, if any, quoting you used in your command.

set -x shows you the already expanded values - which are therefore all literals at that point - and selectively single-quotes them if they contain so-called shell metacharacters (such as spaces) or globbing chars.

Since your glob didn't match anything and was therefore left as-is as a literal, set -x enclosed it in single quotes due to the presence of * chars.

For a robust solution (notably, one that works with filenames with embedded spaces), you need to collect the globbing results in an array first, and pass the array to grep:

# Collect matching paths - note that the path is _unquoted_ to ensure
# that globbing is applied (you can selectively quote _literal_ substrings,
# if needed).
paths=( /logs/XXX/production/.snapshot/ifs-QBA_LS1-VAULT_30Day_2016-06-29_19-01/jboss/pvmk*/xxx-production-xxx*/AppLog/xxx-production-xxx*-apimetrics.log )

# Error out, if nothing matched.
(( ${#paths[@]} > 0 )) || { echo "No matching files." >&2; exit 1; }

# Now, pass the array to `grep`, which will expand it to
# individual filename parameters:
... grep -H '' "${paths[@]}" | ...