Why spark application fail with "executor.CoarseGrainedExecutorBackend: Driver Disassociated"?

Finally I found the reason. It is because Yarn kills the executor (container) because the executor is memory overhead. Just turn up values of spark.yarn.driver.memoryOverhead or spark.yarn.executor.memoryOverhead or both.