Debugging "Managed memory leak detected" in Spark 1.6.0

I found this warn message too, but it was cause by "df.repartition(rePartNum,df("id"))". My df is empty, and the warn message's line equals to rePartNum. version: spark2.4 win10

The short answer is that users are not supposed to see this message. Users are not supposed to be able to create memory leaks in the unified memory manager.

That such leaks happen is a Spark bug: SPARK-11293

---

But if you want to understand the cause of a memory leak, this is how I did it.

1. Download the Spark source code and make sure you can build it and your build works.
2. In TaskMemoryManager.java add extra logging in acquireExecutionMemory and releaseExecutionMemory: logger.error("stack trace:", new Exception());
3. Change all the other debug logs to error in TaskMemoryManager.java. (Easier than figuring out logging configurations...)

Now you will see the full stack trace for all allocations and deallocations. Try to match them up and find the allocations without deallocations. You now have the stack trace for the source of the leak.