java.lang.OutOfMemoryError: Unable to acquire 100 bytes of memory, got 0
In my case replacing the coalesce(1) with repartition(1) Worked.
I believe that the cause of this problem is coalesce(), which despite the fact that it avoids a full shuffle (like repartition would do), it has to shrink the data in the requested number of partitions.
Here, you are requesting all the data to fit into one partition, thus one task (and only one task) has to work with all the data, which may cause its container to suffer from memory limitations.
So, either ask for more partitions than 1, or avoid coalesce() in this case.
Otherwise, you could try the solutions provided in the links below, for increasing your memory configurations:
- Spark java.lang.OutOfMemoryError: Java heap space
- Spark runs out of memory when grouping by key
The problem for me was indeed coalesce().
What I did was exporting the file not using coalesce() but parquet instead using df.write.parquet("testP"). Then read back the file and export that with coalesce(1).
Hopefully it works for you as well.