I'm running a spark application on my IntelliJ IDE as a Maven Project,
I'm trying to create a rowRDD and convert them to dataframe and store it in hdfs.
SPARK VERSION : 1.5.2
SCALA VERSION: 2.10.4
val rowRDD= dataframename.map(_.split("\t")).map(p => Row(p(0),p(1),p(2),p(3)))
From the Spark DataFrame Documentation:
map[R](f: (Row) ⇒ R)(implicit arg0: ClassTag[R]): RDD[R] "Returns a new RDD by applying a function to all rows of this DataFrame."
So when you call
map on a DF, you are mapping over
Row objects which do not have a