sahil desai sahil desai - 1 year ago 100
R Question

How to update to SparkR 2.0.0 package in R

I want to update from

SparkR 1.4.0
SparkR 2.0.0
, but I get the following error:

had non-zero exit status

This is because
SparkR 2.0.0
is not available on CRAN.

Similarly, from
SparkR 1.6.2
SparkR 2.0.0
, we get:

Warning in install.packages :
package ‘SparkR’ is not available (for R version 3.3.0)

How does one install
Spark 2.0.0
from there? (i dont want to install the SparkR because it's already installed as 1.6.2 nut i have to update it as 2.0.0)

Answer Source

SparkR requires not just an R package but an entire Spark backend to be pulled in. When you want to upgrade SparkR, you are upgrading Spark, not just the R package.

Nowadays you may want to refer to the sparklyr package as it makes all of this a whole lot easier.

spark_install(version = "1.6.2")
spark_install(version = "2.0.0")

It also offers more functionality than SparkR.