Can the JVM max heap size be dynamic?
It doesn't. It could, and it probably should:
-Xmx90% // 90% of physical memory
However, a default implicit, 100%, is proabbly not a good idea.
A program written in a non-GC language manages its memory very diligently, it will prune any garbage as soon as possible. It makes sense to allow it to get any memory it requests, assuming it's responsible for prompt garbage disposal.
A GC language is different. It collects garbage only when necessary. As long as there's room, it doesn't care about garbage lingering around. If it could get all the memory it would like to have, it would get all the memory in the computer.
So a GC programmer doesn't have to worry about disposing every piece of garbage any more, but he still have to have a general idea of the tolerable garbage/live object ratio, and instruct GC with -Xmx.
But, is there a way to make that value dynamic?
Literally, no. The max heap size is set at JVM launch time and cannot be increased.
In practice, you could just set the max heap size to as large as your platform will allow, and let the JVM grow the heap as it needs. There is an obvious risk in doing this; i.e. that your application will use all of the memory and cause the user's machine to grind to a halt. But that risk is implicit in your question.
EDIT
It is worth noting that there are various -XX... GC tuning options that allow you to tweak the way that the JVM expands the heap (up to the maximum).
Another possibility is to split your application into 2 parts. The first part of the application does all of the preparation necessary to determine the "size" of the problem. Then it works out an appropriate max heap size, and launches the memory hungry second part of the application in a new JVM.
This only works if the application can sensibly be partitioned as above.
This only works if it is possible to compute the problem size. In some cases, computing the problem size is tantamount to computing the result.
It is not clear that you will get better overall performance than if you just let the heap grow up to a maximum size.
Basically, you can't adapt to various users' hardware using pure Java: that's when a little bit of shell/batch scripting can come in handy.
I do just that on OS X and Linux: I've got a little bash shell script that takes care of finding the correct JVM parameters depending on the hardware the application is run on and then calling the JVM.
Note that if you're providing a desktop Java application, then you may want to use something like izpack to provide your users an installer:
http://izpack.org
I don't know at all if Java Web Start can be used to provide different JVM parameters depending on the user's config (probably not, and JWS really s*cks big time anyway if you plan to provide a professional looking desktop app).