What happens if you specify max heap size greater than available RAM

The easiest way to find out is try it and see.

Edit: There are actually at least two answers to the question. Probably on a 64 bit system, as was mentioned, your app could grow and grow in memory usage and start thrashing. On a 32 bit system the story is a little different because the os is not able to give you that much heap space. For instance, if I run an app on Windows XP with 32 bit java with the command line option -Xmx2000m it will die with a message similar to the following:

Invalid maximum heap size: -Xmx2000m

The specified size exceeds the maximum representable size.

Could not create the Java virtual machine.

In Linux with 32 bit java, I get the following with -Xmx3000m:

Could not create the Java virtual machine.

Error occurred during initialization of VM

Could not reserve enough space for object heap

In Linux with 32 bit java, I get the following with -Xmx6000m

Invalid maximum heap size: -Xmx6000m

The specified size exceeds the maximum representable size.

Could not create the Java virtual machine.

Trying this with 64 bit Java, the JVM does allow you to allocate more memory than there is physical RAM, though if you ask for an extremely large amount of memory, the jvm will again fail with an error.

Only if your -Xms (minimum) is larger than available memory will you get an immediate failure on initialization of the JVM

$>java -Xms100g            #JVM fails to start
Error occurred during initialization of VM Could not
reserve enough space for object heap

If your -Xmx (maximum) is larger than available memory your JVM does initialize since you are not using memory yet

$>java -Xmx100g            #JVM starts up fine
Usage: java [-options] class [args...]
...

If your -Xmx (maximum) is larger than the available memory (total memory to include any virtual memory) you will get a runtime failure if and only if your JVM processes actually tries to use more memory than the machine has.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5feb100000, 927465472, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 927465472 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /some/file/path/hs_err_pid25.log

It wont 'thrash' until it nears your -Xmx limit, but if that limit is above your available memory you will get the above memory allocation error and your program will terminate before thrashing is even considered. (And that is very dramatic!)