Edamame Edamame - 1 month ago 15
Scala Question

Spark losing println() on stdout

I have the following code:

val blueCount = sc.accumulator[Long](0)
val output = input.map { data =>
for (value <- data.getValues()) {
if (record.getEnum() == DataEnum.BLUE) {
blueCount += 1
println("Enum = BLUE : " + value.toString()
}
}
data
}.persist(StorageLevel.MEMORY_ONLY_SER)

output.saveAsTextFile("myOutput")





Then the blueCount is not zero, but I got no println() output! Am I missing anything here? Thanks!

Answer

I was able to work it around by making an UtilityFunction:

object PrintUtiltity {
    def print(data:String) = {
      println(data)
    }
}
Comments