Puneet Chaurasia Puneet Chaurasia - 3 months ago 18
Scala Question

How to convert an Iterable to an RDD

To be more specific, how can i convert a scala.Iterable to a org.apache.spark.rdd.RDD ?

I have an RDD of (String, Iterable[(String, Integer)])
and i want this to be converted into an RDD of (String, RDD[String, Integer]), so that i can apply a reduceByKey function to the internal RDD.

e.g
i have an RDD where key is 2-lettered prefix of a person's name and the value is List of pairs of Person name and hours that they spent in an event

my RDD is :


("To", List(("Tom",50),("Tod","30"),("Tom",70),("Tod","25"),("Tod",15))
("Ja", List(("Jack",50),("James","30"),("Jane",70),("James","25"),("Jasper",15))


i need the List to be converted to RDD so that i can use accumulate each person's total hours spent. Applying reduceByKey and make the result as

("To", RDD(("Tom",120),("Tod","70"))
("Ja", RDD(("Jack",120),("James","55"),("Jane",15))


But i counldn't find any such transformation function. How can i do this ?

Thanks in advance.

Answer

You can achieve this by using a flatMap and reduceByKey. Something like this:

rdd.flatMap{case(key, list) => list.map(item => ((key,item._1), item._2))}
   .reduceByKey(_+_)
   .map{case((key,name),hours) => (key, List((name, hours)))}
   .reduceByKey(_++_)