The "Dzhanibekov effect" - an exercise in mechanics or fiction? Explain mathematically a video from a space station

One can see this effect qualitatively from Newtonian first principles such as $F=ma$ (as opposed to Hamiltonian or Lagrangian principles, such as conservation of energy and angular momentum) by looking at a degenerate case, when one moment of inertia is very small and the other two are very close to each other.

More specifically, consider a thin rigid unit disk, initially oriented in the $xy$ plane and centred at the origin $(0,0,0)$. We make the "spherical cow" hypotheses that this disk has infinitesimal thickness and mass, but infinite rigidity. On this disk, we place heavy point masses of equal mass $M$ at the points $(1,0,0)$ and $(-1,0,0)$ on the $x$ axis, and light point masses of equal mass $m$ at the points $(0,1,0)$ and $(0,-1,0)$ on the $y$ axis. Here $0 < m \ll M$, i.e. $m$ should be viewed as negligible with respect to $M$. (The moments of inertia are then $2m, 2M, 2(m+M)$, though we will not explicitly use these moments in the analysis below.)

We now set up the unstable equilibrium by rotating the disk around the $y$ axis. Thus, the light $m$-masses stay fixed on the $y$-axis, while the heavy $M$-masses rotate in the $xz$-plane. This is in equilibrium: there are no net forces on the $m$-masses, while the rigid disk exerts a centripetal force on the $M$-masses that keeps them moving in a circular motion on the $xz$-plane.

We can view this equilibrium in rotating coordinates, matching the motion of the $M$-masses. (Imagine a camera viewing the disk, rotating around the $y$-axis at exactly the same rate as the disk is rotating.) In this rotating frame, the disk is now stationary (so the $m$-masses are stuck on the $y$-axis at $(0,\pm 1,0)$ and the $M$-masses are stuck on the $x$-axis at $(\pm 1,0,0)$), but there is a centrifugal force exerted on all bodies proportional to the distance to the $y$-axis. The $m$-masses are on the $y$-axis and thus experience no centrifugal force; but the $M$-masses are away from the $y$-axis and thus experience a centrifugal force, which is then balanced out by the centripetal forces of the rigid disk.

Now let us perturb the disk a bit, so that the $m$-masses and $M$-masses are knocked a little bit out of position (but keeping the centre of mass fixed at $(0,0,0)$). In particular, the $m$-masses are knocked away from the $y$-axis and now experience a little bit of centrifugal force. On the other hand, the rigid disk forces the light $m$-masses to remain orthogonal to the heavy $M$-masses, by exerting tension forces between the masses. In the regime where $m$ is negligible compared to $M$, these tension forces will barely budge the heavy $M$ masses (which therefore remain essentially fixed at $(\pm 1,0,0)$ in the rotating frame), so the effect of these tension forces is to constrain the $m$-masses to lie in the $yz$-plane (up to negligible errors which we now ignore). Rigidity also keeps the $m$-masses at a unit distance from the origin, and antipodal to each other, so the $m$-masses are now constrained to be antipodal points on the unit circle in the $yz$-plane. However, other than this, rigidity imposes no further constraints on the location of the $m$-masses, which can then move freely as antipodal points in this unit circle.

The effect of centrifugal force in the rotating frame is now clear: if an $m$-mass (and its antipode) is perturbed to be a little bit off the $y$-axis in this unit circle with no initial velocity, then centrifugal force will nudge it a little further off the $y$-axis, slowly at first but with inexorable acceleration. Eventually it will shoot across the unit circle and then approach the antipode of its previous position. At this point the centrifugal forces act to slow the $m$-masses down, reversing all the previous acceleration, until one ends up with no velocity at a small distance from the antipode. The process then repeats itself (imagine a marble rolling frictionlessly between two equally tall hills, starting from a position very close to the peak of one of the hills).

UPDATE, September 2019: Due to renewed interest in this question, I will expand upon my 2014 comment regarding why the above analysis appears at first glance to also lead to the incorrect conclusion that the disk rotation is also unstable if it one instead rotates around the $x$-axis (so that it is now the $M$-masses that are stationary and the $m$-masses that are rotating), or equivalently if one swaps the location of the $m$-masses and $M$-masses (which we will not do here to try to reduce confusion).

The reason for this is that centrifugal force $F_{\mathrm{Cent}} = -m \Omega \times \Omega \times r$ is only one of two inertial forces that are introduced when one is in a steadily rotating frame. The other inertial force introduced is the Coriolis force $F_{\mathrm{Cor}} = -2 m \Omega \times v$, which acts on moving bodies in the rotating reference frame in a direction orthogonal to the motion (and to the axis of rotation). Strictly speaking, one has to take into account the effect of both inertial forces when performing Newtonian mechanics in a steadily rotating frame. As it turns out, the Coriolis force has a negligible impact on the dynamics when rotating around the $y$-axis, but dominates the dynamics when rotating around the $x$-axis, which is why the preceding discussion is accurate in the former case but not the latter.

In more detail: suppose we are rotating around the $y$-axis as in the above discussion. In the rotating frame of reference, and starting with a configuration slightly out of equilibrium, we have as before that the $m$-masses experience a little bit of centrifugal force and begin to slide away from the $y$-axis and into the rest of the $yz$-plane. When doing so, they will then experience some Coriolis force in a direction parallel to the $x$-axis (which direction it is depends on the orientation of the rotation, as per the right hand rule formula for the cross product). However, due to the rigidity of the disk, as mediated by tension forces within the disk, it is not possible for the $m$-masses to actually move in the $x$-direction without also moving the much heavier $M$-masses. But the magnitude of the Coriolis force is proportional to the small mass $m$ rather than the large mass $M$, so by Newton's law $F=Ma$ for the $M$-masses, the Coriolis force (or more precisely, the tension force produced in response to the Coriolis force) actually barely affects the motion of the $M$-masses, which basically stay put on the $x$-axis, and the $m$-masses therefore remain essentially constrained to the $yz$-plane and cannot actually experience any significant motion in the direction of the Coriolis force. (This is what the sentence in the original explanation regarding how tension forces "barely budge" the $M$-masses was referring too, albeit somewhat obliquely.) The analysis of the original explanation now proceeds as before.

In contrast, suppose that the disk is close to the stable equilibrium state when it rotates around the $x$-axis. Working in a rotating frame around this axis, the $M$-masses are near the $x$-axis of rotation, while the $m$-masses lie near the $y$-axis. As before, the $M$-masses experience centrifugal force and thus begin drifting slightly away from the $x$-axis into the $xz$-plane. But then the Coriolis force on these masses kicks in, which is now proportional to the heavy mass $M$ rather than the light mass $m$. As such, the rigid tension forces connecting the $M$-masses to the much lighter $m$-masses offer very little resistance to the Coriolis force, and the motion of the $M$-mass begins to rotate out of the $xz$-plane, constantly experiencing Coriolis acceleration in a direction orthogonal to its motion. This effect tends to make the $M$-mass rotate in a tight circle, and basically neutralises the net effect of the centrifugal force by rotating any outward motion away from the $x$-axis back into inward motion. The end result is that in the rotating frame, the disk wobbles a bit around its equilibrium state, but does not dramatically depart from it, which is what one would expect from a stable equilbrium (imagine now a marble rolling frictionlessly around the trough of a valley).

In English-speaking physics classes, this is usually called the "Tennis Racket Theorem", because a tennis racket is a good example of a physical object with three widely separated moments of inertia. If you look in this video, you can see the 180 degree flip you describe taking place here on Earth. The interesting difference between Earth and space is that, on Earth, it is difficult to watch an object in free-fall for more than a second, so you only see a fraction of the orbits described in Marcos and Victor's answers; the video I link to looks like about half an orbit. In space, you can see the cycle happen several times.

There are tons of proofs of the tennis racket theorem online, here is one (Wayback Machine).

This question was also asked on physics.SE; the answer there linked to this nice paper.

Take a look at "Rigid Body Stability Theorem" in chapter 15 of Marsden and Ratiu's "Introduction to Mechanics and Symmetry", which basically states that in situations like the one in the video (where, judging by the shape of the rotating object, the principal moments of inertia are mutually distinct), rotation about the middle principal axis is unstable. While the angular momentum of the rigid body is constant in its spatial representation, it is nonconstant with respect to the body's frame. There is a figure following the aforementioned theorem which shows the flow lines for the angular momentum with respect to the body's frame. There are three pairs of antipodal fixed points corresponding to rotation exactly about the principal axes. The "long" and "short" axes' critical points are stable (in the sense of Liapunov), while the "middle" critical points are saddle points, and therefore unstable. Furthermore, there are flow lines coming arbitrarily close to both of the antipodal "middle" critical points. I believe the solutions associated with these flow lines are what you seek.