Why does spark-submit fail with "IllegalArgumentException: Missing application resource."?
tl;dr Remove --jars
option and start over.
java.lang.IllegalArgumentException: Missing application resource.
You missed your...well...Spark application that the message refers to as "application resource".
That's more obvious when you execute spark-submit
and see the different command-line options and their meanings.
./bin/spark-submit
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
That part <app jar | python file | R file>
is what you missed.
To reproduce your issue you can simply execute spark-submit
with --jars
options without specifying the main jar or class of a Spark application.
$ ./bin/spark-submit --jars target/spark-parent_2.11-2.3.0-SNAPSHOT-tests.jar
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:274)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
at org.apache.spark.launcher.Main.main(Main.java:86)
Quoting spark-submit --help
, --jars
is...
--jars JARS Comma-separated list of jars to include on the driver and executor classpaths.
--jars
can be very helpful when a Spark application depends on additional jar files (aka dependencies), i.e. mysql-connect.jar
that you cannot (or most likely don't want to) "assembly" to your uber jar.
--jars
option is added when you've to supply supporting jars to your application jar.
Application resource missing implies that your main jar is missing as you've passed it with --jars
option. It's looking for your main jar and since it couldn't find it, it throws that error.