ulrich ulrich - 4 months ago 47
Scala Question

Importing Spark libraries using Intellij IDEA

I would like to use spark SQL in an Intellij IDEA SBT project.

Even though I have imported the library the code does not seem to import it.

Spark Core seems to be working however.

enter image description here


You can't create a DataFrame from a scala List[A]. You need first to create an RDD[A], and then transform that to a DataFrame. You also need an SQLContext:

val conf = new SparkConf()

val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)

import sqlContext.implicits._

val test = sc.parallelize(List(1,2,3,4)).toDF