a.moussa a.moussa - 1 year ago 64
Scala Question

catch an error on spark

I use a code like this to execute some transformation and to continue the process even if I have bad value:

import scala.util.Try

val data = Array(Seq(1, 2), Seq(3, 4, 5))
val rdd = sc.parallelize(data)
val rdd_test = rdd.flatMap(x => Try(x(2)).toOption)
rdd_test.collect.foreach(println)


this simple code return 5 because in my first line, I don't have any value at third position. My question is how to catch the first line in order to use log4j to log an alert message. I want to say something like:
Index Error.

Do you have any idea?

Answer Source

You could evaluate it with Success and Failure and create the Option that way:

rdd.flatMap{x =>
  val resTry = Try(x(2))
  resTry match {
    case Success(res)  => Some(res)
    case Failure(e) => logger.error("Error: " + Throwables.getStackTraceAsString(e))
      None
    }
}
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download