ATmega328p sine wave generator, frequency capping at 1200Hz

For 1200hz and a 256 lookup table you have 16000000/(256*1200) = 52 cycles between interrupts.

If you count the steps in the interrupt ASM code you're at the rock bottom limit if not below.

In the main loop there is a jump that needs two cycles, if you add nop's the jump will occur less often , that's why you have the tiny improvement.

You can move the interrupt code into the main loop to spare some cycles ( down to three times less) because PUSH's and POP's are slower. Then use nop's to obtain the desired frequency. Disable any interrupt.

There is also one big limitation that is still there , how can you update a 256 step PWM after only 52 cycles? Even if you don't want to reduce the look-up table length many writings to PWM are actually ignored.

Since there is nothing you can do except the value update you might improvise a resistor DAC on the digital port.

Besides what @Dorian says, note that you are operating the PWM timer and the sampling timer at the same frequency. You have one PWM cycle every 256 CPU cycles. If you change the PWM duty cycle more frequently than every 256 CPU cycles, in fast PWM mode you will get glitches/distortions in the output.

To mitigate the problems, in a first step you could add a low-pass (RC) filter at the PWM output to create a sine-like signal of x Hz from a 50% PWM of x Hz, obviating the lookup table. Or use a higher frequency low-pass and reduce the lookup table to, say 4 or 8 entries, reducing the ISR frequency to 4 or 8x the output frequency (instead of 256x) and letting the filter smooth out the transitions between the steps.

As an alternative, you could look into the ATtiny2/4/85 chips which offer a "real" fast PWM off a timer running at up to 64MHz.