Apache Spark upgrade from 1.5.2 to 1.6.0 using homebrew leading to permission denied error during execution
I ran into the same issue and the easiest fix is to set $SPARK_HOME
to /usr/local/Cellar/apache-spark/<your_spark_version>/libexec/
instead.
You can also build from source directly and you can find the instructions here.
Basically just do
git clone https://github.com/apache/spark/`
cd spark
git checkout origin/branch-X.Y
build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
You'll need to set the $SPARK_HOME
to the top level directory of the spark source code.
A ticket on the project page helped:
unset SPARK_HOME && spark-submit
https://github.com/Homebrew/homebrew/issues/48898#issuecomment-180633967