Understanding JVM Memory Allocation and Java Out of Memory: Heap Space

The JVM (or for that matter any process) that wants to allocate memory will call the C runtime 'malloc' function. This function maintains the heap memory of the C runtime. It, in turn, obtains memory from the operating system kernel - the function used for this is platform dependent; in Linux it could be using the brk or sbrk system calls.

Once the memory has been obtained by the JVM, it manages the memory itself, allocating parts of it to the various objects created by the running program.

Virtual memory is handled entirely by the operating system kernel. The kernel manages mapping of physical memory pages to the address space of various processes; if there is less physical memory than is needed by all the processes in the system then the OS Kernel will swap some of it out to disk.

You can't (and don't need to) force processes to use Virtual Memory. It is transparent to your process.

If you are getting 'out of memory' errors, then the causes are likely to be:

1. The JVM limits are being exceeded. These are controlled by various command line arguments and/or properties as you stated in your question

2. The OS may have run out of swap space (or not have any swap space configured to start with). Or some OSs don't even support virtual memory, in which case you have run out of real memory.

3. Most OSs have facilities for the administrator to limit the amount of memory consumed by a process - for example, in Linux the setrlimit system call and/or the ulimit shell command, both of which set limits that the kernel will observe. If a process requests more memory than is allowed by the limits, then the attempt will fail (typically this results in an out of memory message).

How does the JVM actually allocate memory for itself?

For the heap it allocate one large continuous region of memory of the maximum size. Initially this is virtual memory however, over time it becomes real memory for the portions which are used, under control of the OS

How does this relate to how the OS communicates available memory to the JVM?

The JVM has no idea about free memory in the OS.

Or more generally, how does memory allocation for any process actually work?

In general it uses malloc and free.

How does virtual memory come into play?

Initially virtual memory is allocated and this turns into real memory as used. This is normal for any process.

Let's say you have a system with 32GB of physical memory and you allocate all 32GB to your Java process.

You can't. The OS need some memory and there will be memory for other purposes. Even within the JVM the heap is only a portion of the memory used. If you have 32 GB of memory I suggest as 24 GB heap max.

Let's say that your process actually consumes all 32GB of memory,

Say you have 48 GB and you start a process which uses 32 GB of main memory.

how can we enforce the process to use virtual memory instead of running into OOM exceptions?

The application uses virtual memory right from the start. You cannot make the heap too large because if it starts swapping your machine (not just your application) will become unusable.

You can use more memory than you have physical by using off heap memory, carefully. However managed memory must be in physical memory so if you need a 32 GB heap, buy 64 GB of main memory.