Best way to limit number of threads running certain section of code in Java?

This is exactly what java.util.concurrent.Semaphore was designed to do. You create a Semaphore like so:

final int MAX_NOF_THREADS = 5;
final Semaphore mySemaphore = new Semaphore(MAX_NOF_THREADS);

then for the critical area you'd do:

try {
    mySemaphore.aquire(); // This will hang until there is a vacancy
    do_my_critical_stuff();
} finally {
    mySemaphore.release();
}

... as simple as that.

Though, Semaphore is the best choice here (look at the @Bex's answer) if you're careful, it's also possible to use ExecutorService. Just wrap the piece of code you'd like to protect from unlimited concurrent access into a Callable task and submit such tasks to the executor service:

// Task that will be executed
public class MyTask implements Callable<Void> {
    @Override
    public Void call() {
        // Do the work here
        return null;
    }
}

// Service to execute tasks in no more than 5 parallel threads
// Cache it after creation and use when you need to execute a task
int maxThreadsCount = 5;
ExecutorService executor = Executors.newFixedThreadPool(maxThreadsCount);

// Execute a task. It will wait if all 5 threads are busy right now.
executor.submit(new MyTask());

With ExecutorService you can also use Runnable instead of Callable, invokeAll() instead of execute, wait for task completion, cancel tasks, return values from them and do some other useful things.

Java 8 makes it simpler, you can use lambdas instead of defining task classes:

executor.submit(() -> {
    // Do the work here
});