Program does not terminate immediately when all ExecutorService tasks are done

Executors.newCachedThreadPool() uses Executors.defaultThreadFactory() for its ThreadFactory. defaultThreadFactory's javadocs say that "each new thread is created as a non-daemon thread" (emphasis added). So, the threads created for the newCachedThreadPool are non-daemon. That means that they'll prevent the JVM from exiting naturally (by "naturally" I mean that you can still call System.exit(1) or kill the program to cause the JVM to halt).

The reason the app finishes at all is that each thread created within the newCachedThreadPool times out and closes itself after some time of inactivity. When the last one of them closes itself, if your application doesn't have any non-daemon threads left, it'll quit.

You can (and should) close the ExecutorService down manually via shutdown or shutdownNow.

See also the JavaDoc for Thread, which talks about daemon-ness.

I would expect my program/process to stop immediately after all workers are done. But according to my log, it takes another 20-30 seconds until that happens. The workers do not allocate any resources, in fact, they do nothing at the moment.

The problem is that you are not shutting down your ExecutorService. After you submit all of the jobs to the service, you should shutdown the service or the JVM will not terminate unless all of the threads in it are daemon threads. If you do not shutdown the thread-pool then any threads associated with the ExecutorService, again if not daemon, will stop the JVM from finishing. If you've submitted any tasks to a cached thread pool then you will have to wait for the threads to timeout and get reaped before the JVM will finish.

ExecutorService threadPool = Executors.newCachedThreadPool();
for(int i = 0; i < workerCount; i++) {
    threadPool.execute(new Worker());
}
// you _must_ do this after submitting all of your workers
threadPool.shutdown();

Starting the threads as daemon is most likely not what you want to do because your application may stop before the tasks have completed and all of the tasks will be terminated immediately at that time. I just did a quick audit and of the 178 times we use ExecutorService classes in our production code, only 2 of them were started as daemon threads. The rest are properly shutdown.

If you need to force an ExecutorService to stop when the application is exiting then using shutdownNow() with proper handling of the thread interrupt flags is in order.

Basically on an ExecutorService you call shutdown() and then awaitTermination():

ExecutorService taskExecutor = Executors.newFixedThreadPool(4);
while(...) {
   taskExecutor.execute(new MyTask());
}
taskExecutor.shutdown();
try {
  taskExecutor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
  ...
}