Conservative central force and stable orbits

This isn't going to be a rigorous treatment, but you can get an idea of what is going on by considering the effective potential of the orbiting body. If we consider a frame that is corotating with the body then in this frame there is a fictitious centrifugal force pushing the body outwards, and associated with this force there is a potential $L^2/2mr^2$ where $L$ is the (constant) angular momentum. Adding this to the gravitational potential $U(r)$ gives an effective potential:

$$ V_\text{eff} = \frac{L^2}{2mr^2} + U(r) $$

If we take $U(r) = -k/r$ then we get an effective potential looking something like:

If it's a bound orbit this potential always has a minimum. This exists because if the orbiting object approaches the central mass the positive $r^{-2}$ term dominates and pushes it out again, while if the object moves to large $r$ the negative $r^{-1}$ term pulls it in again. Effective the particle oscillates around the minimum of the effective potential so its orbit is always stable.

The problem is that with different gravitational potentials we don't get this stable minimum. For example if $U(r) = -kr^{-2}$ the effective potential is just $\pm Ar^{-2}$ for some constant $A$. With the plus sign the potential always pushes the orbiting object outwards to infinity and with the minus sign the potential pulls it inwards to crash into the central body.

Move to $U(r) = -kr^{-3}$, or any $-kr^{-n}$ for $n \ge 3$, and now we get only a maximum like:

So again any perturbation sends the orbiting object outward to infinity or inwards to crash into the object.

The special feature of the $-kr^{-1}$ potential is that it falls more slowly with distance than the $r^{-2}$ potential due to the fictitious force. That's why only it give a stable orbit. If you allow non-integral powers then any $n \lt 2$ also gives a stable orbit, though we normally consider only integral values of $n$.

Not every central force admits a circular orbit since only an attractive interaction can balance the repulsive centrifugal term. On the other hand, every attractive force has a circular orbit since by an appropriate choice of angular momentum the centrifugal term, $L^2/2\mu r^2$, can be chosen to cancel the attractive interaction, $U(r)$. Finally, attractive potential is not a sufficient for the existence of stable orbits.

The circular orbit of radius $r_0$ is stable if and only if $U_{\mathrm{eff}}(r_0)$ corresponds to a minimum of the effective potential, $$U_{\mathrm{eff}}(r)=\frac{L^2}{2\mu r^2}+U(r).$$ This implies that the second derivative of $U_{\mathrm{eff}}$ at $r_0$ must be positive. Hence, $$U''(r_0)>-\frac{3L^2}{\mu r_0^4}.\tag1$$

For circular orbits, the radial effective force (which includes interaction and centrifugal forces) vanishes and then $U_{\mathrm{eff}}(r_0)'=0$. Thus $$r_0^3=\frac{L^2}{\mu U'(r_0)}.$$ Plugging this into Eqn (1) we obtain the following condition $$\frac{U''(r_0)}{U'(r_0)}+\frac{3}{r_0}>0,$$ for stable orbits. As an example, assuming an attractive potential with a power law $U=kr^{n}$, the last equation gives us that is has stable orbits only for $n>-2$.

It is worth mentioning that there are different concepts of stability regarding orbital motion. The one assumed here, for which the above result holds, says that a circular orbit is stable if it remains bounded under small perturbations (a bounded orbit is one whose radius is limited by $r_{\mathrm{min}}\leq r\leq r_{\mathrm{max}}$). A different and also common concept is Lyapunov Stability. In this case, among all power law central forces, only the harmonic oscillator gives stable orbits.