programmatically setting max java heap size

Not with any Hotspot JVM. The JVM heap parameters can only be specified on the command line, and are then fixed for the lifetime of the JVM.

With Hotspot Java implementations, the only way to "change" the heap size of an application is to relaunch it in a new JVM with different command line parameters.

(I vaguely recall that JVMs from some other vendors do allow some heap parameters to be changed in a running JVM. Perhaps someone can clarify.)