gsjunior86 gsjunior86 - 3 months ago 28
Scala Question

How to correctly sum an integer inside .map function in Spark?

I'm new to Scala and Spark, trying to create a pair-like RDD in Spark, assigning an Int as key for each line and summing +1.

val mapUrls ={
var cont = 0
x =>
cont += 1

the problem is that somehow, the cont variable goes back to 1 after certain time.

What am i doing wrong?


Is this is what you want?

Your code just cannot work correctly. Remember that Spark is distributed framework and there is no shared memory. Each task has updates its own cont.