Vinay Vinay - 7 months ago 30
Bash Question

File input getting consumed inside the while loop

I'm reading through a lookup file and performing a set of actions for each line in the file. However the while loop only reads the first line in the file and exits. Here's the current code that I have.

sql_from_lkp $lookup

function sql_from_lkp {
lkp=$1
while read line; do
sql_from_columns ${line}
echo ${line}
done < ${lkp}
}

function sql_from_columns {
table_name=$1
table_column_info_file=${table_name}_columns
line_count=`cat $table_info_file | wc -l`
....
}


By selectively commenting the code, I found that if I comment the
line_count
line, the while loop goes through every line in the file and works fine. So the input is getting consumed by the
cat
statement.

I've checked other answers and understood that
ssh
usually consumes the file inputs inside
while
loops if -n option is not used. But not sure how to fix this case. Need some help.

Answer

You've mistyped a variable name: $table_info_file should be $table_column_info_file.

If you correct that, your problem will go away.


By referring to a non-extant variable - the mistyped $table_info_file - you're essentially executing cat | wc -l (no filename argument passed to cat) in sql_from_columns(), which makes cat read from stdin.

Therefore - after having read the 1st line in the while loop - the cat command in sql_from_columns() consumes the entire rest of your input (< ${lkp}), which is why the while loop exits after the 1st iteration.


Generally,

  • You should double-quote all your variable references so as not to subject their values to word-splitting and globbing.

  • Bash won't allow you to call functions before they're defined, so as presented in your question, your code fundamentally couldn't work.

  • While the legacy `...` syntax for command substitutions is still supported, it has pitfalls that can be avoided with the modern $(...) syntax.

  • A more efficient way to count lines is to pass the input file to wc -l via < rather than via cat and a pipeline (wc also accepts filename operands directly, but it then prints the input filename after the counts).

    • Incidentally, you probably would have caught your mistyped variable reference more easily had you done that, as Bash would have reported an ambiguous redirect error in the absence of a filename following <.

Here's a reformulation that addresses all the issues:

function sql_from_lkp {
    lkp=$1
    while read line; do
        sql_from_columns "${line}"
        echo "${line}"
    done < "${lkp}"
}

function sql_from_columns {
    table_name=$1
    table_column_info_file=${table_name}_columns
    line_count=$(wc -l < "$table_column_info_file")
    # ...
}

sql_from_lkp "$lookup"

Note that I've only added double quotes where strictly needed to make the command robust; it wouldn't hurt to add them whenever a parameter (variable) is referenced.