Where is the photon getting energy to deviate from its path?

Momentum is a vector. That vector is changed, not its magnitude. Otherwise we'd see a slit changing the wavelength of the photons. The change does cause a force on the barrier structure, but the barrier is not moving. If the barrier moved, there would be an energy transfer between photon and barrier, and the photon wavelength would change.

BTW, an interesting side effect of this fact is that if the barrier moved toward the source of photons, the resulting photons would have more energy.