SMPS: What is current mode instability (aka “sub-harmonic oscillation”)?

Rather than going into the mathematics of this, it's quite easy to see this graphically. Consider a peak current mode controller operating at <50% duty cycle. Then you can see below that perturbing the system results in the perturbation decaying and the system returning to steady-state operation.

But at >50% duty-cycle the system does not return to steady state operation. Instead, it enters a "sub-cycle oscillation" mode.

Simple as that.

See source for a more detailed explanation.

You can find a good explanation of all this in this paper from TI:

Current Mode Control

Basically subharmonic oscillation is not a small signal instability, it's a large signal phenomenon in peak current mode control that occurs when the duty cycle is greater than 50%. (Though under certain circumstances it can occur below 50% duty cycle as well.) By adding a ramp signal to the output of the error amp it is possible to stabilize a peak current mode converter for all duty cycles, but the control loop no longer behaves exactly like a current mode converter.