septra septra - 4 months ago 108
Linux Question

Apache Spark upgrade from 1.5.2 to 1.6.0 using homebrew leading to permission denied error during execution

I just upgraded Spark from 1.5.2 to 1.6.0 using Homebrew and reset my

environment variable to
. Now while executing pyspark, it gives a permission denied error.

If I go into the earlier 1.5.2 installation directory and execute pyspark from there, it runs fine. But running pyspark from the 1.6.0 installation directory fails with this permission denied error.

/usr/local/Cellar/apache-spark/1.6.0/bin/ line 2: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/ Permission denied

/usr/local/Cellar/apache-spark/1.6.0/bin/ line 2: exec: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/ cannot execute: Undefined error: 0

What could be causing this?


Worked by doing unset SPARK_HOME && pyspark (credit to noli below)

Wanted to get the SparkContext running in Ipython, turned out to be quiet straightforward.

unset SPARK_HOME IPYTHON=1 pyspark