data princess data princess - 5 months ago 46
R Question

How to resolve "sql(sqlContext...)' is deprecated" warning in SparkR

I'm building a new version of some old code using SparkR. Upon a block like this

hiveContext <- sparkRHive.init(sc)
hive_db = 'our_database'
db <- sql(hiveContext, paste0("use ", hive_db))

I'm told that
'sparkRHive.init' is deprecated. Use 'sparkR.session' instead.
So, okay, fine, I now have:

hiveContext <- sparkR.session(sc)
hive_db = 'our_database'
db <- sql(hiveContext, paste0("use ", hive_db))

This runs, but now Spark warns
'sql(sqlContext...)' is deprecated. Use 'sql(sqlQuery)' instead.
I'm at a loss for what kind of input it's expecting here and would like to resolve this. Has anyone figured out what to do here?

Answer Source

Since Spark 2.0 sql and the number of other functions (like createDataFrame) dont require SQLContext instance. Just:

sql(paste0("use ", hive_db))

Internally this will use getSparkSession to retrieve a session object.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download