Ananth Duari Ananth Duari - 1 year ago 160
Java Question

Spark Streaming with Twitter - No output streams registered, so nothing to execute

I'm trying to execute a spark streaming example using Twitter. This is my code

public static void main (String.. args) {

SparkConf conf = new SparkConf().setAppName("Spark_Streaming_Twitter").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(2));
JavaSQLContext sqlCtx = new JavaSQLContext(sc);

String[] filters = new String[] {"soccer"};

JavaReceiverInputDStream<Status> receiverStream = TwitterUtils.createStream(jssc,filters);



But I'm getting the following exception

Exception in thread "main" java.lang.AssertionError: assertion failed: No output streams registered, so nothing to execute
at scala.Predef$.assert(Predef.scala:179)
at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:158)
at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:416)
at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:437)
at org.learning.spark.TwitterStreamSpark.main(

Any suggestion how to fix this issue?

Answer Source

When an output operator is called, it triggers the computation of a stream.

Without output operator on DStream no computation is invoked. basically you will need to invoke any of below method on stream

saveAsObjectFiles(prefix, [suffix])
saveAsTextFiles(prefix, [suffix])
saveAsHadoopFiles(prefix, [suffix])

you can also first apply any transformations and then output functions too if required.