gsjunior86 gsjunior86 - 1 year ago 85
Scala Question

How to correctly sum an integer inside .map function in Spark?

I'm new to Scala and Spark, trying to create a pair-like RDD in Spark, assigning an Int as key for each line and summing +1.

val mapUrls ={
var cont = 0
x =>
cont += 1

the problem is that somehow, the cont variable goes back to 1 after certain time.

What am i doing wrong?

Answer Source

Is this is what you want?

Your code just cannot work correctly. Remember that Spark is distributed framework and there is no shared memory. Each task has updates its own cont.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download