How to debug a scala based Spark program on Intellij IDEA
First define environment variable like below
export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=7777
Then create the Debug configuration in Intellij Idea as follows
Rub-> Edit Configuration -> Click on "+" left top cornor -> Remote -> set port and name
After above configuration run spark application with spark-submit or sbt run and then run debug which is created in configuration. and add checkpoints for debug.
If you're using the scala plugin and have your project configured as an sbt project, it should basically work out of the box.
Go to Run
->Edit Configurations...
and add your run configuration normally.
Since you have a main
class, you probably want to add a new Application
configuration.
You can also just click on the blue square icon, to the left of your main
code.
Once your run configuration is set up, you can use the Debug feature.
It is similar to the solution provided here: Debugging Spark Applications. You create a Remote debug run configuration in Idea and pass Java debug parameters to the spark-submit command. The only catch is you need to start the remote debug config in Idea after triggering the spark-submit command. I read somewhere that a Thread.sleep just before your debug point should enable you to do this and I too was able to successfully use the suggestion.
I've run into this when I switch between 2.10 and 2.11. SBT expects the primary object to be in src->main->scala-2.10 or src->main->scala-2.11 depending on your version.