DomainsFeatured DomainsFeatured - 2 months ago 11
Linux Question

Will Running A GNU Parallel Command With Variables Cause Conflict?

I think this comes down to me not fully understanding how the processes of GNU Parallel are divided. I looked up if using variables with GNU Parallel would conflict and I found very little to nothing online.

If I'm using a command like the following with parallel:

echo "$textdata" | parallel -j5 cat | for line in file; do var1=$(echo $line); var2=$(echo "$line" | grep -A 1); var3=$(echo "$line" | somecommand); echo "$var1" "$var2" "$var3"; done


Would it conflict by overriding each other when using the variable? Or, does it run in different processes and the same variable can be used?

In other words, would $var1, $var2 or $var3 get confused between different processes running in parallel?

Answer

For anyone else wanting to know if running variables in GNU parallel will overlap, simply just take the command, such as:

echo "$textdata" | parallel -j5 cat | for line in file; do var1=$(echo $line); var2=$(echo "$line" | grep -A 1); var3=$(echo "$line" | somecommand); echo "$var1" "$var2" "$var3"; done 

And run the command as a script.sh:

for line in file; do var1=$(echo $line); var2=$(echo "$line" | grep -A 1); var3=$(echo "$line" | somecommand); echo "$var1" "$var2" "$var3"; done 

Thus is would be:

echo "$textdata" | parallel -j5 /script.sh

Hope this helps anyone working with parallel and variables.

Comments