alex alex - 4 months ago 13
Bash Question

Bash wait does not work

Part of my code:

for i in "${r_arr[@]}"
do
${parallel_dir}parallel -j${cap} --semaphore --semaphorename a2_bootstrap_semaphore_${rnd} a2_exec ${i} ${temp_zmat_folder_path}${temp_zmat_folder}/ ${outs_folder}
done

wait
elapsed_time=$(($SECONDS - $start_time))

rm -rf "${temp_zmat_folder_path}${temp_zmat_folder}"
echo "Cleanup done."
echo "`date +%d-%m-%y` `date +%T`: All processing finished in ${elapsed_time} seconds!"


As you can see I'm starting a number of tasks using GNU Parallel. The problem is that cleanup (
rm
) is executed before last of the parallel tasks is even started. How to prevent such behaviour?

Answer

As per their example, you should use this command to wait for all the started jobs to complete

parallel --semaphore --wait

Example (From man page): The command sem is an alias for parallel --semaphore.

for i in *.log ; do
    echo $i
    sem -j10 gzip $i ";" echo done
  done
  sem --wait
Comments