How to connect with JMX remotely to Spark worker on Dataproc

As Rick Mortiz pointed out, the issue was conflicting ports for the executor jmx.

Setting:

-Dcom.sun.management.jmxremote.port=0

yields a random port, and removed the errors from Spark. To figure out which port it ends up using do:

netstat -alp | grep LISTEN.*<executor-pid>/java

which lists the currently open ports for that process.