Anaadih.pradeep Anaadih.pradeep - 3 months ago 12
Scala Question

Group By error 33 : wrong number of parameter

I am kind of new to Spark and programming too and i am not able to understand how to deal with Rdd of type

rdd.RDD[(Int, Iterable[Double])] = ShuffledRDD[10] at groupByKey
. I am little bit interested in learning groupByKey in spark and i have a filtered RDD

scala> p.first
res11: (Int, Double) = (1,299.98)


I go the above result after applying GroupByKEy instead of reduceByKey now i have rdd of type (Int, Iterable[Double]) and i want to get result like (Int , sum (Double)).
I have tried this but got the error.

scala> val price = g.map((a,b) => (a, sum(b)))
<console>:33: error: wrong number of parameters; expected = 1
val price = g.map((a,b) => (a, sum(b)))


Please suggest and help me in this to understand it

Answer

g.mapValues(_.sum), which is short for g.map { case (k, v) => (k, v.sum) }