I am using spark streaming in scala, where I am working with streaming twitter data. I have the following code:
val ssc = new StreamingContext(new SparkConf(), Seconds(5))
val tweets = TwitterUtils.createStream(ssc, None)
val user = tweets.map(x=> x.getText())
val lang = tweets.map(x=> x.getLang())
[error] /home/user/Lab1.1/Twitterstats.scala:103: value getLang is not a member of twitter4j.Status
[error] val lang = tweets.map(x=> x.getLang())
[error] one error found
spark-streaming-twitter uses Twitter4j.
getLang() is only supported since version 3.0.6 of Twitter4J. If you are using a version 1.5.2 (or lower) of spark-streaming-twitter you won't be able to call
getLang() because it uses version 3.0.3 of twitter4j. Since 1.6.0 version 4.0.4 is supported as wel as the
So you could upgrade spark-streaming-twitter to 1.6.0 or higher. Or you could use another 3rd party library to detect the language of your tweets.