p3zo p3zo - 11 months ago 104
Scala Question

Spark streaming `if(!rdd.partitions.isEmpty)` not working

I'm trying to create a dStream from a kafka server and then do some transformations on that stream. I have included a catch for if the stream is empty (

); however, even when no events are being published to the kafka topic, the
statement is never reached.

val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)

stream.foreachRDD { rdd =>
if(!rdd.partitions.isEmpty) {

val message = rdd.map((x$2) => x$2._2).collect().toList.map(parser)

val val = message(0)

} else println("empty stream...")



Is there an alternative statement I should use to check if the stream is empty when using
rather than

Answer Source

Use RDD.isEmpty instead of RDD.partitions.isEmpty which adds a check to see if the underlying partition actually has elements:

stream.foreachRDD { rdd =>
  if(!rdd.isEmpty) {
    // Stuff

The reason RDD.partitions.isEmpty isn't working is that there exists a partition inside the RDD, but that partition itself is empty. But from the view of partitions which is an Array[Partition], it isn't empty.