In Submitting Applications in the Spark docs, as of 1.6.0 and earlier, it's not clear how to specify the --jars argument, as it's apparently not a colon-separated classpath not a directory expansion.
The docs say "Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes."
Question: What are all the options for submitting a classpath with
--jars in the spark-submit script in $SPARK_HOME/bin? Anything undocumented that could be submitted as an improvement for docs?
/usr/local/spark/bin/spark-submit --class jpsgcs.thold.PipeLinkageData ---jars=local:/usr/local/spark/jars/groovy-all-2.3.3.jar,local:/usr/local/spark/jars/guava-14.0.1.jar,local:/usr/local/spark/jars/jopt-simple-4.6.jar,local:/usr/local/spark/jars/jpsgcs-core-1.0.8-2.jar,local:/usr/local/spark/jars/jpsgcs-pipe-1.0.6-7.jar /usr/local/spark/jars/thold-0.0.1-1.jar
In this way it worked easily.. instead of specifying each jar with version separately..
#!/bin/sh # build all other dependent jars in OTHER_JARS JARS=`find ../lib -name '*.jar'` OTHER_JARS="" for eachjarinlib in $JARS ; do if [ "$eachjarinlib" != "APPLICATIONJARTOBEADDEDSEPERATELY.JAR" ]; then OTHER_JARS=$eachjarinlib,$OTHER_JARS fi done echo ---final list of jars are : $OTHER_JARS echo $CLASSPATH spark-submit --verbose --class <yourclass> ... OTHER OPTIONS --jars $OTHER_JARS,APPLICATIONJARTOBEADDEDSEPERATELY.JAR