Monitor non-heap memory usage of a JVM

Your heap is actually using 6.5 GB of virtual memory (this may include the perm gen)

You have a bunch of threads using 64 MB stacks. Not clear why some are and others are using the default 1 MB.

The total is 9.3 million KB of virtual memory. I would only worry about the resident size.

Try using top to find the resident size of the process.

You may find this program useful

    BufferedReader br = new BufferedReader(new FileReader("C:/dev/gistfile1.txt"));
    long total = 0;
    for(String line; (line = br.readLine())!= null;) {
        String[] parts = line.split("[- ]");
        long start = new BigInteger(parts[0], 16).longValue();
        long end = new BigInteger(parts[1], 16).longValue();
        long size = end - start + 1;
        if (size > 1000000)
            System.out.printf("%,d : %s%n", size, line);
        total += size;
    }
    System.out.println("total: " + total/1024);

---

Unless you have a JNI library using the memory, my guess is you have lots of threads which each have their own stack space. I would check the number of threads you have. You can reduce the maximum stack space per thread, but a better option might be to reduce the number of threads you have.

The off heap memory is by definition unmanaged, so it is not easily "tuned" as such. Even tuning the heap is not simple.

The default stack size on 64-bit JVMs is 1024K so 700 threads will be using 700 MB of virtual memory.

You shouldn't confuse virtual memory sizes for resident memory sizes. Virtual memory on a 64-bit application is almost free and it's only the resident size you should worry about.

The way I see it you have 9.3 GB total.

• 6.0 GB heap.
• 128 MB perm gen
• 700 MB stacks.
• < 250 shared libraries
• 2.2 GB of unknown (I suspect virtual memory not resident memory)

The last time some one had this problem they had a lot more threads than they though they should. I would check the maximum number of threads you had as it is the peak which determines the virtual size. e.g. was it closer to 3000?

---

Hmmm each of these pairs is a thread.

7f0cffddf000-7f0cffedd000 rw-p 00000000 00:00 0 
7f0cffedd000-7f0cffee0000 ---p 00000000 00:00 0

and these suggest you have slightly less than 700 threads now.....

While mr Lawrey answered in great detail where and how you're loosing memory, i believe it can be useful to have some specific steps like (do this do that and you'll know where your java memory goes)...

His answer didn't really help me with my similar off heap memory usage, and in my case it was definitely not a threading problem.

Application that uses just 30mb of heap and seems perfectly healthy, consumed 700% more off heap for no good reason. Eventually linux would kill it and i could not tell why, no heap dump analysis helped with eclipse memory analyser...

The tool that helped me out is called jxray. It is not free (nothing good is) but it has a trial.

1. head to https://jxray.com/download and get the tool
2. get a heap dump (yes i know you want off heap memory but just do it)
3. generate a report ./jxray.sh /path/to/dump

It will create a html file report right next your memory dump which will have to the point summary of whats going where and where is your problem.

In my case it looks like this.

Then you can zoom in into problem and see where it comes from. Apparently the tool is smart enough to look into allocated sizes of direct byte buffers to realize that your application is using far more than you have in your heap dump.

In my case i got lazy and used okhttp for a simple long polling http request that is the whole purpose of this small application. Apparently it leaked memory very very slowly, and my application would die once every few weeks. I got rid of okhttp, upgraded java to 13 and used native http client now everything is working fine, and i have one less crap library in my classpath.

I also recommend using it on your healthy applications, pretty sure you will find some interesting facts that you did not know about them )