What happens to orbits at small radii in general relativity?
I was so surprised by the types of graphs I saw for this that I felt compelled to add an answer. As mentioned in the Wikipedia article I linked to, there are two radii that are of interest in addition to the Schwarzschild radius $r_s$. Those radii are
- the "Innermost Stable Circular Orbit" (ISCO) $ r_{outer} \approx2a^{2}/r_{s}$ and
- the "Innermost Bound Circular Orbit" (IBCO) $r_{inner} \approx 3/2 r_{s}$.
It is clear that the inner radius is clearly defined for the BH, but the outer radius depends on the specific orbital parameters. By definition the Kerr-Spin-parameter $a=J/(mc)$, where $J$ is the angular momentum of the central black hole and $m$ is its mass.* This, along with most info here, assumes the particle orbiting the BH has a small mass by comparison.
Here is one paper's graphs, covered by New Scientist. These patterns reflect the general pattern of zoom-whirl behavior, which might also be called Homoclinic orbits. As far as I can tell, these happen when an orbit traverses the ISCO but does not traverse the IBCO. I take it that (I am assuming that) if an orbit does not traverse the ISCO it has a shape like what I posted in the question, which I think are formally called Keplerian orbits.
Can some of these orbit in the small radius (or "zoom") more than once before making back out to the whirl? As it turns out, yes. An orbit can zoom many times before whirling, up to infinity, as per this paper.
For this orbit of infinite zooms, it would get infinitely close to the IBCO. In fact, that shines a lot of light on the nature of the IBCO as an unstable but balanced orbit point. No orbit can exist closer than the IBCO because it will just spiral inward, like a toilet bowl. In fact, there are two ways to dismiss non-orbits, all initial conditions either orbit, fall into the event horizon eventually, or they escape.
Kerr black holes, two-body problem, and other complications
Obviously we should expect more orbit possibilities if the black hole is spinning. Everything I've cited so far seems to be specific to orbits of small masses (relative to the BH) and non-rotating BHs. A lot of the combinations are apparently still being studied.
Under some conditions, a transition to chaos happens (reference: presentation from the periodic table of BH orbits paper researchers). Exactly what those conditions are is something I'm still unclear on, as a result of me not fully understanding this material.
In the case of a rotating black hole, it would seem that you can even orbit one way, and then stop and start orbiting the other way. Reference: http://www.lsw.uni-heidelberg.de/users/mcamenzi/Kerr_Black_Holes.pdf
I have a hard time imagining how the above picture is consistent with the hyperbolic orbits I see in Wikipedia, as it does not show this direction change at all. Maybe someone else can shed light on the difference and how they can both be true, if that is the case.
Also, if the particle isn't in the plane of the equator of the Kerr black hole, it will move up and down in the direction of the axis of rotation, giving it fully 3D dynamics. Here are some examples.
You can imagine, this is just scratching the surface. Combine that graph with the crazy zoom-whirl graphs, maybe even make it a 2-body problem, and it would make for some pretty fun paths.
In some conditions, the orbit becomes a Homoclinic tangle, which I won't even post a picture of because it just looks like a big knot.
* clarification from further questions in Expression for distance of closest approach in Schwarzschild Geodesics
What does this mean?
It means that there won't be any (periodic) orbit anymore; the answer to your title question is therefore that it will cease to exist. The value of $r$ will just monotonically decrease. Obviously, when it falls below the event horizon, there's no way for the particle to return outside the black hole i.e. to values of $r$ greater than the event horizon's. The particle will end up in the singularity.
Why is the stability transition point further than the Schwarzschild radius?
These two points have different values of $r$ because they are defined by different conditions. The event horizon is the boundary beneath which one cannot escape outside, whatever he does; he may try to use his jets to escape as quickly as he can but it won't be enough to escape if he's beneath the event horizon.
The minimal orbit radius is the minimal value of $r$ beneath which one cannot escape if he is only allowed to freely fall. Clearly, if one doesn't resist, it's easier for the gravitational field to swallow him, so the region from which the singularity is an inevitable fate in this case is larger.
Why does this graph show an unstable point beyond the Schwarzschild radius?
I just explained why the critical values of $r$ beneath which one can no longer oscillate are inevitably outside the event horizon, so it's the same question as the second question answered here. There can't ever be periodic orbits inside the black hole (less than event horizon) because this would contradict the fact that an observer inside is inevitably dragged towards the singularity.
For elliptical orbits that come very close to the event horizon, is there some kind of orbital decay?
There are no eliptical orbits near the event horizon anymore. This is the main point that this whole material is about although because you didn't know the answer when you were writing the question, it may be justified that you added lots of confusing questions caused by your assuming wrong answers to the previous ones.
How is energy conserved?
Energy is perfectly conserved in all these considerations. Like always in similar mechanical exercises, even in non-relativistic mechanics, the decrease of kinetic energy is compensated by the increase of potential energy and vice versa. However, the formulae for the potential and kinetic energy have a new, nonlinear dependence on $r$ which is why it's no longer true that all the trajectories are simple conics. One must say that even in Newton's gravity, the conics character of all trajectories was a sort of a coincidence, one that doesn't appear for any other potential than $K/r$.
Note that even in Newton's mechanics, it's untrue that all trajectories are periodic. With too high a velocity, the trajectories are parabolic or hyperbolic.
In short, do all orbits (with GR effects) look like the precession shown above, or is there another shape that we see if it gets closer to the Schwarzschild radius?
All orbits qualitatively look like precession but, as discussed in every single question above, there are no back-and-forth orbits for certain initial conditions. So for these initial conditions, going too close to the event horizon, the trajectories will qualitatively look like "spirals".