Parag Kuhikar Parag Kuhikar - 1 month ago 17
Scala Question

How to run twitter popular tags of Spark streaming using scala?

I am new to Spark so please guide.

There are lots of example available related to Spark streaming using Scala.

You could check it out from https://github.com/apache/incubator-spark/tree/master/examples/src/main/scala/org/apache/spark/streaming/examples.

I want to run TwitterPopularTags.scala.

I am not able to set the twitter login details for this example.

http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html#linking-with-spark-streaming

I am successfully run the network count example.

But when i execute

./run-example org.apache.spark.streaming.examples.TwitterPopularTags local[2]

then it will show me authentication failure issue...

I set twitter login details before initializing string context in TwitterPopularTags.scala like

System.setProperty("twitter4j.oauth.consumerKey", "####");
System.setProperty("twitter4j.oauth.consumerSecret", "##");
System.setProperty("twitter4j.oauth.accessToken", "##");
System.setProperty("twitter4j.oauth.accessTokenSecret", "##");


Please guide.

Answer

Put the file "twitter4j.properties" into the Spark root directory (e.g. spark-0.8.0-incubating) before you run the Twitter examples.

twitter4j.properties:

oauth.consumerKey=***
oauth.consumerSecret=***
oauth.accessToken=***
oauth.accessTokenSecret=***

Worked for me on Mac with the Scala examples.