sarthak sarthak - 1 year ago 97
Scala Question

Error in Spark Streaming code in scala

I am using spark streaming in scala, where I am working with streaming twitter data. I have the following code:

val ssc = new StreamingContext(new SparkConf(), Seconds(5))
val tweets = TwitterUtils.createStream(ssc, None)

val user => x.getText())
val lang => x.getLang())

I am getting the following error:

[error] /home/user/Lab1.1/Twitterstats.scala:103: value getLang is not a member of twitter4j.Status
[error] val lang => x.getLang())
[error] ^
[error] one error found

What is wrong with the above code? Could someone please help.

Answer Source

spark-streaming-twitter uses Twitter4j. getLang() is only supported since version 3.0.6 of Twitter4J. If you are using a version 1.5.2 (or lower) of spark-streaming-twitter you won't be able to call getLang() because it uses version 3.0.3 of twitter4j. Since 1.6.0 version 4.0.4 is supported as wel as the getLang() function.

So you could upgrade spark-streaming-twitter to 1.6.0 or higher. Or you could use another 3rd party library to detect the language of your tweets.

(possible duplicate)