D. Müller D. Müller - 1 year ago 224
Scala Question

Apache Zeppelin 0.6.1: Run Spark 2.0 Twitter Stream App

I have a cluster with Spark 2.0 and Zeppelin 0.6.1 installed. Since the class

is moved from Spark project to Apache Bahir, I can't use the TwitterUtils in my Zeppelin notebook anymore.

Here the snippets of my notebook:

Dependency loading:


DepInterpreter(%dep) deprecated. Remove dependencies and repositories through GUI interpreter menu instead.
DepInterpreter(%dep) deprecated. Load dependency through GUI interpreter menu instead.
res1: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Dependency@4793109a

And the Spark part:

import org.apache.spark.streaming.twitter
import org.apache.spark.streaming._
import org.apache.spark.storage.StorageLevel
import scala.io.Source
import scala.collection.mutable.HashMap
import java.io.File
import org.apache.log4j.Logger
import org.apache.log4j.Level
import sys.process.stringSeqToProcess
import org.apache.spark.SparkConf

// ********************************* Configures the Oauth Credentials for accessing Twitter ****************************
def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {...}

// ***************************************** Configure Twitter credentials ********************************************
val apiKey = ...
val apiSecret = ...
val accessToken = ...
val accessTokenSecret = ...
configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)

// ************************************************* The logic itself *************************************************
val ssc = new StreamingContext(sc, Seconds(2))
val tweets = TwitterUtils.createStream(ssc, None)
val twt = tweets.window(Seconds(60))

When I try to run the Spark part of the notebook after importing the dependency, I get the following exception:

<console>:44: error: object twitter is not a member of package org.apache.spark.streaming
import org.apache.spark.streaming.twitter

What am I doing wrong here? Bahir documentation also uses the
import org.apache.spark.streaming.twitter._
command, see http://bahir.apache.org/docs/spark/2.0.0/spark-streaming-twitter/

Answer Source

Well, dep is not exactly stable and since it is deprecated anyway why not use supported methods? If you don't won't to modify neither Spark nor Zeppelin configuration files you can add dependencies to the interpreter configuration (I omitted properties for clarity):

enter image description here