How to know what is the reason for ClosedChannelExceptions with spark-shell in YARN client mode?

Reason is association with yarn cluster may be lost due to the Java 8 excessive memory allocation issue: https://issues.apache.org/jira/browse/YARN-4714

You can force YARN to ignore this by setting up the following properties in yarn-site.xml

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

Thanks to simplejack, Reference from Spark Pi Example in Cluster mode with Yarn: Association lost

Personally I resolved this by increasing yarn.nodemanager.vmem-pmem-ratio as suggested in the Jira ticket by Akira Ajisaka:

<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>5</value>
</property>

I have built another answer which depends whether you are using spark client or cluster mode.

• In cluster mode it failed when I specified Driver Memory --driver-memory to be 512m. (The default setting requested 2GB of am resources (This consists of driver memory + Overhead requested for Application Master) which was enough)
• In client mode the setting that mattered was spark.yarn.am.memory as by default this requested only 1024m for the AM which is too little as Java 8 requires a lot of virtual memory. > 1024m seemed to be working.

Answer is described here