Anthony Anthony - 2 months ago 8
Scala Question

Is it possible to create an RDD in sbt console?

I'm working in SBT console and have imported the following

scala> import org.apache.spark._
scala> val sc = new SparkContext("local", "shell")
scala> import org.apache.spark.rdd._


I would like to make an
RDD[Myclass]
in the console. Is that possible?

scala> case class MyClass(foobar: String, something: String)


The reason I want to do this is to play around with RDD and create sample data that I can test with. Rather than reading a textFile or reading huge data from HDFS or S3

Answer

You can use sc.parallelize and give that a List of your objects.