A.Dumas A.Dumas - 3 months ago 43
Java Question

Where can I set the spark.eventLog.enabled?

I want to test out the machine learning library Mllib form spark and set up a maven project with the dependency in eclipse. When I try to compile this example I get the following exception: `ERROR SparkContext: Error initializing SparkContext.

ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:371)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
at testing.JavaLogisticRegressionSummaryExample.main(JavaLogisticRegressionSummaryExample.java:33)


There has been a suggestion to set
spark.eventLog.enabled
too false but where do I set this value?

Answer

If you are running in eclipse as java program then you can set master URL like below. When you create a jar with dependency and run in spark using spar-submit then you can pass master url in command. Please refer Spark documentation.

SparkSession spark = SparkSession
  .builder()
  .appName("JavaLogisticRegressionSummaryExample")
  .config("spark.sql.warehouse.dir", "/file:C:/temp")//workaround for issue https://issues.apache.org/jira/browse/SPARK-15899
  .master("local[2]")
  .getOrCreate();
Comments