toofrellik toofrellik - 2 months ago 15
Scala Question

sc.textFile cannot resolve symbol at textFile

I am using IntelliJ for scala
below is my code:

import org.apache.spark.SparkConf
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object XXXX extends App{

val sc = new SparkConf()
val DataRDD = sc.textFile("/Users/itru/Desktop/XXXX.bz2").cache()
}


my build.sbt file contains below:

name := "XXXXX"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "1.5.2" %"provided",
"org.apache.spark" %% "spark-sql" % "1.5.2" % "provided")


Packages are downloaded, but still i see error "cannot resolve symbol at textFile" Am i missing any library dependencies

Answer

You haven't initialised your SparkContext.

val sparkConf = new SparkConf()
val sc = new SparkContext(sparkConf)

val dataRDD = sc.textFile("/Users/itru/Desktop/XXXX.bz2").cache()
Comments