How to force Mathematica to clean up the cache

This is also not an answer (I think you found the culprit and it needs to be solved by WRI) but a suggestion for a workaround. The idea is to run the memory leaking code in an extra kernel and restart that kernel every once in a while. Using the parallel functionality of current Mathematica releases this is pretty simple and by choosing an appropriate value for the number of parallel kernels you could even take advantage of parallelism if desired.

Here is a simple example based on your minimal working example which demonstrates the idea and shows that the master kernel will not accumulate any memory:

LaunchKernels[1];
Print[MemoryInUse[]];
calcsPerKernel = 3;
Do[
   With[{fname = "plot-" <> ToString[i] <> ".png"},
     ParallelEvaluate[
       Export[
         FileNameJoin[{$HomeDirectory, "Desktop", fname}],
         ListStreamPlot[
           Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000]
         ]
       ];
       Print[MemoryInUse[]];
     ]
   ];
   If[Mod[i, calcsPerKernel] == 0,
     CloseKernels[];
     ClearSystemCache[];
     LaunchKernels[1];
   ];
   ,
  {i, 10}
];
CloseKernels[];
ClearSystemCache[];
Print[MemoryInUse[]];

The idea for a more general and flexible approach is as follows:

  • use LaunchKernels to launch the desired number of parallel kernels
  • use ParallelSubmit to pass the computations (e.g. per file) to the parallel kernel(s)
  • use WaitNext to get the result of a computation from the other kernel(s).
  • use CloseKernels[kernelid] and LaunchKernels to restart a/the parallel kernels

As it is very difficult to completely avoid memory leaks in sufficient complex Mathematica code I think this is a technique that will be helpful in many situations. We are using this approach regularly for computations that run for several days and have found that it works very reliable. Unlike some other parts of Mathematica the parallel toolkit seems to work very stable and reliable even on a larger scale.


Not an answer, just extended clarification

There seem to be many types of caching related leaks here.

I can get one by just doing this:

$HistoryLength = 0;
Do[Table[RandomReal[], 251], 200]
ClearSystemCache[];
MemoryInUse[]

Calling that repeatedly shows a steady increase with brief pauses.

This doesn't cause a leak:

Do[Table[RandomReal[], 249], 200]
ClearSystemCache[];
MemoryInUse[]

Which shows that something is leaking in the internal auto-compilation.

These are yet distinct from what happens if you apply ListPlot to that. This causes a leak every call:

Do[ListPlot@Table[RandomReal[], 251], 5]
ClearSystemCache[];
MemoryInUse[]

While this continues not to leak:

Do[ListPlot@Table[RandomReal[], 249], 5]
ClearSystemCache[];
MemoryInUse[]

On the other hand if we use ListStreamPlot the leaks seem to be at top-level as this:

Do[ListStreamPlot@
    Table[RandomReal[{}, {2, 2}], 2];, 2];
ClearSystemCache[];
MemoryInUse[]

Leaks every single time

Tags:

Memory

Recent Posts