Why certain rotations are unstable? (Euler Equations)

There is another nice way of seeing this mathematically. It is not too hard to show that in the body frame, there are two conserved quantities: the square of the angular momentum vector $$ L^2 = L_1^2 + L_2^2 + L_3^2 $$ and the rotational kinetic energy, which works out to be $$ T = \frac{1}{2}\left( \frac{L_1^2}{I_1} + \frac{L_2^2}{I_2} + \frac{L_3^2}{I_3} \right). $$ (Note that the angular momentum $\vec{L}$ itself is not conserved in the body frame; but its square does happen to be a constant.)

We can then ask the question: For given values of $L^2$ and $T$, what are the allowed values of $\vec{L}$? It is easy to see that $L^2$ constraint means that $\vec{L}$ must lie on the surface of a sphere; and it is almost as easy to see that the $T$ constraint means that $\vec{L}$ must also lie on the surface of a given ellipsoid, with principal axes $\sqrt{2TI_1} > \sqrt{2T I_2} > \sqrt{2T I_3}$. Thus, the allowed values of $\vec{L}$ must lie on the intersection of a sphere and an ellipsoid. If we hold $L^2$ fixed and generate a bunch of these curves for various values of $T$, they look like this:

Note that for a given value of $L^2$, an object will have its highest possible kinetic energy when rotating around the axis with the lowest moment of inertia, and vice versa.

Suppose, then, that an object is rotating around the axis of its highest moment of inertia. If we perturb this object so that we change its energy slightly (assuming for the sake of argument that $L^2$ remains constant), we see that the vector $\vec{L}$ will now lie on a relatively small curve near its original location. Similarly, if the object is rotating around its axis of lowest inertia, $\vec{L}$ will stay relatively close to its original value when perturbed.

However, the situation is markedly different when the object is rotating about the intermediate axis initially (the third red point in the diagram above, on the "front side" of the sphere. The contours of slightly perturbed $T$ near this point do not stay near the intermediate axis; they wander all over the sphere. There is therefore nothing keeping $\vec{L}$ from wandering all over this sphere if we perturb the object slightly away from rotating about this axis; which implies that an object rotating about its intermediate axis is unstable.

There's an alternative to @MichaelSeifert method which uses angular momentum and moments of inertia: it is to deal with the vector $\vec\omega$ directly as we are interested in the evolution of this vector.

One can express the kinetic energy and length squared of $\vec L$ as \begin{align} T&=\frac{1}{2}\left(I_1\omega_1^2+I_2\omega_2^2+I_3\omega_3^2\right)\\ \vec L\cdot\vec L&=I_1^2\omega_1^2+I_2^2\omega_2^2+I_3^2\omega_3^2 \end{align} These two equations define distinct ellipsoids in $\omega$-space. As there are three components to $\vec\omega$ but only two constraints, the constraints are not enough to completely determine $\vec \omega$. Rather, the evolution of $\vec\omega$ must lie on the line intersection of these two ellipses, as illustrated by the black line in the figure. (The green ellipse is constant $\vec L\cdot \vec L$ and the yellow one constant $T$.)

This intersection here is roughly the shape of a flattened banana, with the long side of the banana in the $\omega_2$ direction.

Thus, during the evolution, the tip of $\vec \omega$ can move along this intersection while also keeping the energy and $\vec L\cdot\vec L$ constant. The figure illustrates the case where $$ I_1=4, \quad I_2=3/2, \quad I_3=1 $$ with $T=5/2$ and $\vert \vec L\vert=2.45$.

You can see from the figure that, if we start on the intersection at the top, the evolution of $\omega_3$ is quite restricted in range (and cannot change sign), that the evolution of $\omega_1$ is also quite restricted, but that the component $\omega_2$ is much larger (basically the entire length of the flattened banana). Hence, qualitatively speaking, the rotation about the middle axis unstable because $\omega_2$ need not stay near its initial value.

(The same argument works for the components $L_k=\omega_k I_k$ in MichaelSeifert's figure except that, in his case, $L_1$ would not change sign but $L_3$ could change sign near the hyperbolic fixed point.)

My physics professor helped us visualize this with a tennis racket.

Spinning with the axis along the handle is stable.

Spinning in the plane of the racket is stable. John McEnroe flipped his rackets like that a lot.

Spinning in the other direction, like how you would swing a racket, is unstable. No matter how carefully you flip the racket, it also spins in the other direction before it comes back to your hand.

In case you don't have access to a racket to play with: https://www.youtube.com/watch?v=4dqCQqI-Gis