ulrich ulrich - 1 year ago 279
Scala Question

Importing Spark libraries using Intellij IDEA

I would like to use spark SQL in an Intellij IDEA SBT project.

Even though I have imported the library the code does not seem to import it.

Spark Core seems to be working however.

enter image description here

Answer Source

You can't create a DataFrame from a scala List[A]. You need first to create an RDD[A], and then transform that to a DataFrame. You also need an SQLContext:

val conf = new SparkConf()

val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)

import sqlContext.implicits._

val test = sc.parallelize(List(1,2,3,4)).toDF
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download