animal animal - 3 months ago 24
Scala Question

Not able to see RDD contents

i am using scala to create an RDD but when i am trying to see the contents of RDD i am getting below results

MapPartitionsRDD[25] at map at <console>:96


I want to see the contents of RDD how can i see that ?

below is my scala code:

object WordCount {
def main(args: Array[String]): Unit = {
val textfile = sc.textFile("/user/cloudera/xxx/File")
val word = textfile.filter(x => x.length > 0).map(_.split('|'))
println(word)
}
}

Answer

You need to provide an output transformation (action). e.g. use RDD.collect:

object WordCount {
   def main(args: Array[String]): Unit = {
     val textfile = sc.textFile("/user/cloudera/xxx/File")
     val word = textfile.filter(x => x.length >  0).map(_.split('|'))
     word.collect().foreach(println)
   }
}

If you have an Array[Array[T]], you'll need to flatten before using foreach:

word.collect().flatten.foreach(println)
Comments