Guforu Guforu - 3 months ago 26
Scala Question

sbt file doesn't recognize the spark input

I try to execute Scala code in Spark. The example of the code and

build.sbt
file is possible to find here.

I have one difference to this example. I use already the version
2.0.0
of Spark (I have already download this version local and defined path in .bashrc file). Now, I have modified also my
build.sbt
file and set the version to
2.0.0

After that I have the error message.

Case 1:
I just executed the code of
SparMeApp
like is given in the link. I got the error message, that I have to define
setMaster
function.

16/09/05 19:37:01 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration


Case 2:
I define
setMaster
function with different arguments. I have got next error messages:

Input:
setMaster("spark://<username>:7077)
or
setMaster("local[2]")

Error:

[error] (run-main-0) java.lang.ArrayIndexOutOfBoundsException: 0
java.lang.ArrayIndexOutOfBoundsException: 0


(this error means that my string is empty)

In other cases just error:
16/09/05 19:44:29 WARN


StandaloneAppClient$ClientEndpoint: Failed to connect to master <...>
org.apache.spark.SparkException: Exception thrown in awaitResult


Additional I have only a little experience in Scala and in sbt. So probably my sbt is configutred false.... Can somebody, please, tell me the right way?

Answer

This is how your minimal build.sbt will look :

name := "SparkMe Project"

version := "1.0"

scalaVersion := "2.11.7"

organization := "pl.japila"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"

And here is your SparkMeApp object :

object SparkMeApp{
  def main(args: Array[String]) {
    val conf = new SparkConf()
      .setAppName("SparkMe Application")
    .setMaster("local[*]")
    val sc = new SparkContext(conf)

    val fileName = args(0)
    val lines = sc.textFile(fileName).cache

    val c = lines.count
    println(s"There are $c lines in $fileName")
  }
}

execute it like :

$ sbt "run [your file path]"

Comments