Creating too many threads in Java

It's not Java that's creating too many threads; you are!

Don't create an executor each time you call a function. If you have 100 collections with 100 elements each, you will create 10,000 threads - that is very resource-consuming... And pointless.

ExecutorService executor = Executors.newFixedThreadPool(threadNum);

You have, most likely, 8 cores - just create one executor with 8 threads and use it everywhere. Your code will work faster and your application will consume less, much much fewer resources.

Familiarize yourself with this code review singleton executor question. You may be able to use that solution in your application.

By using ExecutorService executor = Executors.newFixedThreadPool(threadNum); you are creating a new thread pool for each call of getValuesForKeyFromMaps. So when your keySet contains 100 entries you'll end up with 100 pools with 10-15 threads each. Keep one thread-pool as an instance- or class-variable and use it whenever needed.