ulrich ulrich - 1 month ago 21
Scala Question

Importing Spark libraries using Intellij IDEA

I would like to use spark SQL in an Intellij IDEA SBT project.

Even though I have imported the library the code does not seem to import it.

Spark Core seems to be working however.

enter image description here

Answer

You can't create a DataFrame from a scala List[A]. You need first to create an RDD[A], and then transform that to a DataFrame. You also need an SQLContext:

val conf = new SparkConf()
  .setMaster("local[*]")
  .setAppName("test")

val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)

import sqlContext.implicits._

val test = sc.parallelize(List(1,2,3,4)).toDF
Comments