data princess data princess - 21 days ago 6
R Question

How to resolve "sql(sqlContext...)' is deprecated" warning in SparkR

I'm building a new version of some old code using SparkR. Upon a block like this

hiveContext <- sparkRHive.init(sc)
hive_db = 'our_database'
db <- sql(hiveContext, paste0("use ", hive_db))


I'm told that
'sparkRHive.init' is deprecated. Use 'sparkR.session' instead.
So, okay, fine, I now have:

hiveContext <- sparkR.session(sc)
hive_db = 'our_database'
db <- sql(hiveContext, paste0("use ", hive_db))


This runs, but now Spark warns
'sql(sqlContext...)' is deprecated. Use 'sql(sqlQuery)' instead.
I'm at a loss for what kind of input it's expecting here and would like to resolve this. Has anyone figured out what to do here?

Answer Source

Since Spark 2.0 sql and the number of other functions (like createDataFrame) dont require SQLContext instance. Just:

sql(paste0("use ", hive_db))

Internally this will use getSparkSession to retrieve a session object.