user1615666 user1615666 - 20 days ago 5
Scala Question

spark-submit missing command line arugments

in Scala + Spark, I use spark-submit to run a job with arugments. when tried to print out argument, args.foreach(println), it will skip "PROD" and only output "/tmp" and "A-1" .

Any ideas ? Thanks

spark-submit \
--class "com.Main" \
--master yarn-client \
--num-executors 4 \
--driver-memory 16g \
--executor-cores 10 \
--executor-memory 6g \
--jars main.jar PROD /tmp A-1 2>&1 | tee ./log


Thanks.

Answer

With --jars flag you said to spark you want to transfer main.jar to your cluster (I assume main.jar is your spark application).

When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas.

Spark probably assumed PROD as your application so he skipped this arg.

Try this instead:

spark-submit \
--class "com.Main" \
--master yarn-client \
--num-executors 4 \
--driver-memory 16g \
--executor-cores 10 \
--executor-memory 6g \
main.jar PROD /tmp A-1 2>&1 | tee ./log

if you have still problems with arguments (weird characters like slash, pipeline ...) you can try to use quotes: main.jar PROD "/tmp" "A-1 2>&1 | tee ./log", and in main it should be:

args(0) -- PROD
args(1) -- /tmp
args(2) -- A-1 2>&1 | tee ./log
Comments