septra septra - 5 months ago 155
Linux Question

Apache Spark upgrade from 1.5.2 to 1.6.0 using homebrew leading to permission denied error during execution

I just upgraded Spark from 1.5.2 to 1.6.0 using Homebrew and reset my

SPARK_HOME
environment variable to
/usr/local/Cellar/apache-spark/1.6.0
. Now while executing pyspark, it gives a permission denied error.




If I go into the earlier 1.5.2 installation directory and execute pyspark from there, it runs fine. But running pyspark from the 1.6.0 installation directory fails with this permission denied error.

/usr/local/Cellar/apache-spark/1.6.0/bin/load-spark-env.sh: line 2: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/load-spark-env.sh: Permission denied


/usr/local/Cellar/apache-spark/1.6.0/bin/load-spark-env.sh: line 2: exec: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/load-spark-env.sh: cannot execute: Undefined error: 0


What could be causing this?

Answer

Worked by doing unset SPARK_HOME && pyspark (credit to noli below)

Wanted to get the SparkContext running in Ipython, turned out to be quiet straightforward.

unset SPARK_HOME IPYTHON=1 pyspark