Guforu Guforu - 1 year ago 99
Scala Question

sbt file doesn't recognize the spark input

I try to execute Scala code in Spark. The example of the code and

file is possible to find here.

I have one difference to this example. I use already the version
of Spark (I have already download this version local and defined path in .bashrc file). Now, I have modified also my
file and set the version to

After that I have the error message.

Case 1:
I just executed the code of
like is given in the link. I got the error message, that I have to define

16/09/05 19:37:01 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration

Case 2:
I define
function with different arguments. I have got next error messages:



[error] (run-main-0) java.lang.ArrayIndexOutOfBoundsException: 0
java.lang.ArrayIndexOutOfBoundsException: 0

(this error means that my string is empty)

In other cases just error:
16/09/05 19:44:29 WARN

StandaloneAppClient$ClientEndpoint: Failed to connect to master <...>
org.apache.spark.SparkException: Exception thrown in awaitResult

Additional I have only a little experience in Scala and in sbt. So probably my sbt is configutred false.... Can somebody, please, tell me the right way?

Answer Source

This is how your minimal build.sbt will look :

name := "SparkMe Project"

version := "1.0"

scalaVersion := "2.11.7"

organization := "pl.japila"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"

And here is your SparkMeApp object :

object SparkMeApp{
  def main(args: Array[String]) {
    val conf = new SparkConf()
      .setAppName("SparkMe Application")
    val sc = new SparkContext(conf)

    val fileName = args(0)
    val lines = sc.textFile(fileName).cache

    val c = lines.count
    println(s"There are $c lines in $fileName")

execute it like :

$ sbt "run [your file path]"

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download