Harshit Harshit - 1 year ago 71
Scala Question

Spark jobs finishes but application takes time to close

Running spark job using scala, as expected all jobs are finishing up on time , but somehow some INFO logs are printed for 20-25 minutes before job stops.

Posting few UI screenshot which can help to undestand the problem .

  1. Following is time taken by 4 stages :

Time taken by 4 stages

  1. Following is time between consecutive job ids
    time between consecutive job ids

I dont understand why there is so much time spent in between both job ids.

Following is my code snippet:

val sc = new SparkContext(conf)
for (x <- 0 to 10) {
val zz = getFilesList(lin);
val links = zz._1
val path = zz._2
lin = zz._3
val z = sc.textFile(links.mkString(",")).map(t => t.split('\t')).filter(t => t(4) == "xx" && t(6) == "x").map(t => titan2(t)).filter(t => t.length > 35).map(t => ((t(34)), (t(35), t(5), t(32), t(33))))
val way_nodes = sc.textFile(way_source).map(t => t.split(";")).map(t => (t(0), t(1)));
val t = z.join(way_nodes).map(t => (t._2._1._2, Array(Array(t._2._1._2, t._2._1._3, t._2._1._4, t._2._1._1, t._2._2)))).reduceByKey((t, y) => t ++ y).map(t => process(t)).flatMap(t => t).combineByKey(createTimeCombiner, timeCombiner, timeMerger).map(averagingFunction).map(t => t._1 + "," + t._2)

Some more followup : spark-1.4.1 saveAsTextFile to S3 is very slow on emr-4.0.0

Answer Source

I ended up upgrading my spark version and issue was resolved .

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download