Java "Could not reserve enough space for object heap" even though there is enough RAM

OpenVZ & Memory

The failcnt is going up on privvmpages, so your container is unable to allocate any more virtual memory space from the host:

root@server: ~ # cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
            privvmpages               6005601              6291447              6291456              6291456                 >233<
            physpages                 4635460              6291456              6291456              6291456                    0
            vmguarpages                     0                    0              6291456  9223372036854775807                    0
            oomguarpages              1529376              2144671              6291456  9223372036854775807                    0

Note that virtual memory != physical memory. Processes can allocate up to somewhere around the addressable amount of virtual memory (32bit ~ 2G - 4G, 64bit 8 TB - 256 TB) but that doesn't mean physical memory pages are being used ( a page being a 4KB chunk of memory).

physpages is the number of physical memory pages your container can use.
oomguarpages is the guaranteed memory pages the container will receive when the host is memory constrained.
privvmpages is the number of virtual memory pages your container can use
vmguarpages is the guaranteed amount of virtual memory in the same way

Java

Oracle Java will always allocate one contiguous chunk of virtual memory. Running java with no arguments on a box results in 5M of real memory used (RSS), but 660M of VM space allocated (VSZ):

  PID COMMAND                        VSZ   RSS
20816 java                        667496  4912 

Looking at the memory segments for the java process in it's smaps file shows a chunk of about 500MB allocated, the rest is memory mapped files and normal java stuff.

On a system that's been up for a while the available VM space becomes fragmented as processes use/free parts of it. A grep Vmalloc /proc/meminfo will give you VmallocChunk which is the largest free chunk currently available. If this is low, the system will try and allocate more when java requests it, after all it's virtually unlimited on a 64bit box.

Fix

Tell your host to configure privvmpages and vmguarpages much higher. There's no need for them to be the same as physical memory as that impacts the way linux memory works

You might be able to work around the problem temporarily by dropping your file cache echo 1 > /proc/sys/vm/drop_caches but that's only temporary.

You can limit the chunk of memory java tries to allocate at run time with a minimum Xms or while running with maximum Xmx. Running java with these options on my machine:

java -Xms10M -Xmx10M

reduces the total virtual size to 140MB or so with only a 10MB contiguous chunk for the java heap allocated.

In our case it helped to constrain the heap size that the VM tries to reserve on startup.

For example on the command line:

export _JAVA_OPTIONS='-Xms64M -Xmx128m'

or for Tomcat in [TOMCAT_HOME]/bin/setenv.sh

#!/bin/sh

JAVA_OPTS="-Xms64M -Xmx256M"

(Note the extra leading underscore on the command line.)

The explanation of our provider is:

• the java VM calculates an initial heap size on startup
• this calculation is based upon the RAM available.
• some java installations on virtual private servers (VPS) fail to reflect the memory constrains of the VPS when calculating the initial heap size. Instead they base their calculation upon the memory size of the hosting system. Which may lead to an amount of free memory that is not available on the VPS.
• when providing an initial heap size via JAVA_OPTS the VM uses these values and does not try to calculate on its own.

Probably you need to change the default java heap size, try out these arguments:

-Xms<size>        set initial Java heap size
-Xmx<size>        set maximum Java heap size
-Xss<size>        set java thread stack size

for example:

java -Xms64m -Xmx1512m

This thread could be helpful

Also check the ulimit (ulimit provides control over the resources available to the shell and to processes started by it):

ulimit -a