Growing resident memory usage (RSS) of Java Process
I ran in to the same problem. This is a known problem with glibc >= 2.10
The cure is to set this env variable
export MALLOC_ARENA_MAX=4
IBM article about setting MALLOC_ARENA_MAX https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en
Google for MALLOC_ARENA_MAX or search for it on SO to find a lot of references.
You might want to tune also other malloc options to optimize for low fragmentation of allocated memory:
# tune glibc memory allocation, optimize for low fragmentation
# limit the number of arenas
export MALLOC_ARENA_MAX=2
# disable dynamic mmap threshold, see M_MMAP_THRESHOLD in "man mallopt"
export MALLOC_MMAP_THRESHOLD_=131072
export MALLOC_TRIM_THRESHOLD_=131072
export MALLOC_TOP_PAD_=131072
export MALLOC_MMAP_MAX_=65536
It's also possible that there is a native memory leak. A common problem is native memory leaks caused by not closing a ZipInputStream/GZIPInputStream.
A typical way that a ZipInputStream is opened is by a call to Class.getResource/ClassLoader.getResource and calling openConnection().getInputStream() on the java.net.URL instance or by calling Class.getResourceAsStream/ClassLoader.getResourceAsStream. One must ensure that these streams always get closed.
Some commonly used open source libraries have had bugs that leak unclosed java.util.zip.Inflater or java.util.zip.Deflater instances. For example, Nimbus Jose JWT library has fixed a related memory leak in 6.5.1 version. Java JWT (jjwt) had a similar bug that was fixed in 0.10.7 version. The bug pattern in these 2 cases was the fact that calls to DeflaterOutputStream.close() and InflaterInputStream.close() do not call Deflater.end()/Inflater.end() when an Deflater/Inflater instance is provided. In those cases, it's not enough to check the code for streams being closed. Every Deflater/Inflater instances created in the code must have handling that .end() gets called.
One way to check for Zip*Stream leaks is to get a heap dump and search for instances of any class with "zip", "Inflater" or "Deflater" in the name. This is possible in many heap dump analysis tools such as Yourkit Java Profiler, JProfiler or Eclipse MAT. It's also worth checking objects in finalization state since in some cases memory is released only after finalization. Checking for classes that might use native libraries is useful. This applies to TLS/ssl libraries too.
There is an OSS tool called leakchecker from Elastic that is a Java Agent that can be used to find the sources of java.util.zip.Inflater instances that haven't been closed (.end() not called).
For native memory leaks in general (not just for zip library leaks), you can use jemalloc to debug native memory leaks by enabling malloc sampling profiling by specifying the settings in MALLOC_CONF environment variable. Detailed instructions are available in this blog post: http://www.evanjones.ca/java-native-leak-bug.html . This blog post also has information about using jemalloc to debug a native memory leak in java applications. There's also a blog post from Elastic featuring jemalloc and mentioning leakchecker, the tool that Elastic has opensourced to track down problems caused by unclosed zip inflater resources.
There is also a blog post about a native memory leak related to ByteBuffers. Java 8u102 has a special system property jdk.nio.maxCachedBufferSize to limit the cache issue described in that blog post.
-Djdk.nio.maxCachedBufferSize=262144
It's also good to always check open file handles to see if the memory leak is caused by a large amount of mmap:ed files. On Linux lsof can be used to list open files and open sockets:
lsof -Pan -p PID
The report of the memory map of the process could also help investigate native memory leaks
pmap -x PID
For Java processes running in Docker, it should be possible to execute the lsof or pmap command on the "host". You can find the PID of the containerized process with this command
docker inspect --format '{{.State.Pid}}' container_id
It's also useful to get a thread dump (or use jconsole/JMX) to check the number of threads since each thread consumes 1MB of native memory for its stack. A large number of threads would use a lot of memory.
There is also Native Memory Tracking (NMT) in the JVM. That might be useful to check if it's the JVM itself that is using up the native memory.
The jattach tool can be used also in containerized (docker) environment to trigger threaddumps or heapdumps from the host. It is also able to run jcmd commands which is needed for controlling NMT.