Grant Herman Grant Herman - 1 month ago 16
Scala Question

Build issues for apache spark

I have been having a sbt dependency issue when I try to build my apache spark project. I have Apache Spark 1.3.1.

My .sbt file is this:

name := "Transaction"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1"

resolvers ++= Seq(
"Akka Repository" at "http://repo.akka.io/releases/",
"Spray Repository" at "http://repo.spray.cc/")


And I keep getting this error:

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.10;1.3.1: not found


I have looked all over and this seems to be a persistent issue, but no one has really solved it.

Thanks for your help!

Answer

I actually figured it out. You just need to put "provided"after the spark version.

name := "Transaction Fraud"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1" % "provided"

resolvers ++= Seq(
"Akka Repository" at "http://repo.akka.io/releases/",
"Spray Repository" at "http://repo.spray.cc/")
Comments