How to set the maximum memory usage for JVM?

You shouldn't have to worry about the stack leaking memory (it is highly uncommon). The only time you can have the stack get out of control is with infinite (or really deep) recursion.

This is just the heap. Sorry, didn't read your question fully at first.

You need to run the JVM with the following command line argument.

-Xmx<ammount of memory>

Example:

-Xmx1024m

That will allow a max of 1GB of memory for the JVM.

use the arguments -Xms<memory> -Xmx<memory>. Use M or G after the numbers for indicating Megs and Gigs of bytes respectively. -Xms indicates the minimum and -Xmx the maximum.