Why is OpenJDK 11 Java garbage collector *decreasing* free memory in this sample program?

The answer is: The GC of Java 11 (when explicitly called e.g. via System.gc()) can reduce the used memory of the Java process (in Java known as totalMemory).

In Java 8 the default garbage collector was not able to reduce the used memory of the Java process. Memory occupied by the Java process was never ever released. Only if you switch to the G1GC garbage collector (option '-XX:+UseG1GC') Java 8 is able to reduce the used memory of the Java process (if you manually call System.gc()).

The "free memory" is the memory occupied by the Java process but that is currently not used. Therefore if you perform a garbage collection and the memory occupied by Java is reduced the amount of free memory is reduced, too.

Therefore the common way to calculate the "free memory" of a Java process is to use

Runtime r = Runtime.getRuntime();
long free = r.maxMemory() - r.totalMemory() + r.freeMemory();

This way is independent of the current memory occupied by the Java process.

I think what you are seeing is the JDK deciding that your heap is way too big for your application's needs and returning chunks of it to the operating system, thus reducing your Java heap size and as a side effect also the free/unused heap memory.