eyeOfTheStorm eyeOfTheStorm - 1 year ago 151
R Question

SparklyR removing a Table from Spark Context

Would like to remove a single data table from the Spark Context ('sc'). I know a single cached table can be un-cached, but this isn't the same as removing an object from the sc -- as far as I can gather.

library(sparklyr)
library(dplyr)
library(titanic)
library(Lahman)

spark_install(version = "2.0.0")
sc <- spark_connect(master = "local")

batting_tbl <- copy_to(sc, Lahman::Batting, "batting")
titanic_tbl <- copy_to(sc, titanic_train, "titanic", overwrite = TRUE)
src_tbls(sc)
# [1] "batting" "titanic"

tbl_cache(sc, "batting") # Speeds up computations -- loaded into memory
src_tbls(sc)
# [1] "batting" "titanic"

tbl_uncache(sc, "batting")
src_tbls(sc)
# [1] "batting" "titanic"


To disconnect the complete sc, I would use
spark_disconnect(sc)
, but in this example it would destroy both "titanic" and "batting" tables stored inside of sc.

Rather, I would like to delete e.g., "batting" with something like
spark_disconnect(sc, tableToRemove = "batting")
, but this doesn't seem possible.

Answer Source
dplyr::db_drop_table(sc, "batting")

I tried this function and it seems work.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download