Avoiding GeoServer/Java out of heap space error?

I've had the same error message on a Windows Server with GeoServer using Jetty. After some research I found the file C:\Program Files (x86)\GeoServer 2.10.1\wrapper\wrapper.conf where I edited these two lines:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=16

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=128

Seems to have done the trick...

I found a hint to this file here: http://osgeo-org.1560.x6.nabble.com/Geoserver-heap-size-permgen-space-configuration-td5145590.html

You can increase Memory Allocation in the following ways

(performance settings in the Java virtual machine (JVM) for your container)

-Xmx256M -Xms48m

"Allocates extra memory to your server. By default, JVM will use only 64MB of heap. If you’re serving just vector data, you’ll be streaming, so having more memory won’t increase performance. If you’re serving coverages, however, JAI will use a disk cache. -Xmx256M allocates 256MB of memory to GeoServer (use more if you have excess memory). It is also a good idea to configure the JAI tile cache size (see the Server Config page in the Web Administration Interface section) so that it uses 75% of the heap (0.75). -Xmx48m will tell the virtual machine to grab a 48MB heap on startup, which will make heap management more stable during heavy load serving."

-XX:MaxPermSize=128m

"Increases the maximum size of permanent generation (or “permgen”) allocated to GeoServer to 128MB. Permgen is the heap portion where the class bytecode is stored. GeoServer uses lots of classes, and it may exhaust that space quickly, leading to out of memory errors. This is especially important if you’re deploying GeoServer along with other applications in the same container, or if you need to deploy multiple GeoServer instances inside the same container."

http://docs.geoserver.org/stable/en/user/production/container.html

Changing Java Heap Memory for Geoserver

sudo vi /usr/share/tomcat7/bin/catalina.sh

line 3

export CATALINA_OPTS="-Xms2048m -Xmx8g"

Change the "-Xmx2048m" to desired value, as Xmx is case sensitive, you can use "g" for GB or "m" for MB

After the changes are saved, remember to restart tomcat:

sudo services tomcat7 restart