"Bad substitution" when submitting spark job to yarn-cluster
It is caused by hdp.version
not getting substituted correctly. You have to set hdp.version
in the file java-opts
under $SPARK_HOME/conf
.
And you have to set
spark.driver.extraJavaOptions -Dhdp.version=XXX
spark.yarn.am.extraJavaOptions -Dhdp.version=XXX
in spark-defaults.conf
under $SPARK_HOME/conf
where XXX is the version of hdp.
If you are using spark with hdp, then you have to do the following things:
Add these entries in $SPARK_HOME/conf/spark-defaults.conf
spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)
Create a file called java-opts
in $SPARK_HOME/conf
and add the installed HDP version to that file like this:
-Dhdp.version=2.2.0.0-2041 (your installed HDP version)
To figure out which hdp version is installed, please run this command in the cluster:
hdp-select status hadoop-client