Does the counter of an Atmega328 still count inside interrupt handler?

The counter continues counting unless you set the necessary control bits to stop it. The interrupt handler won't stop it.

The counter will keep counting in the interrupt handler routine just as @DavidMolony has indicated.

There is another thing to keep in mind when the interrupt handler is dynamically updating something like the next next counting period. Lets say that your operational mode is to just let the timer continue to count from 0x0000 to 0xFFFF back to 0x0000 and continue like that. If you adjust the next output compare value on the fly by the amount needed to the next interrupt then there is no need to be concerned about the offset of the interrupt service routine. That is as long as the time to the next output compare interrupt is much longer than the time to process the current interrupt handler.