Classificating the isometries of the Hyperbolic Plane

Here's a way to approach this problem from the linear algebra point of view. And in this answer I'm going to side-step the issue of orientation reversing isometries, which are not represented by Möbius transformation, instead by anti-Möbius transformations. So, I will explain how to use linear algebra to classify orientation preserving (o.p.) isometries, equivalently to classify Möbius transformations.

The classification of o.p. isometries is invariant under conjugacy. To see some examples of this from a synthetic point of view (i.e. no linear algebra), consider an o.p. isometry $\phi$, another o.p. isometry $\psi$, and the conjugate o.p. isometry $\psi \phi \psi^{-1}$.

First, if $\phi$ has a fixed point $x$ then $\psi \phi \psi^{-1}$ also has a fixed point, namely $\psi(x)$. Proof: $$(\psi \phi \psi^{-1})(\psi(x)) = \psi \phi(x) = \psi(x) $$ Furthermore, if $\phi$ rotates by an angle $\theta$ around $x$ then $\psi$ also rotates by an angle $\theta$ around $\phi(x)$. Proof: take a ray $R$ based at $x$, and the angle from $R$ to $\phi(R)$ at $x$ equals $\theta$; so the ray from $\psi(R)$ to $\psi(\phi(R))$ at $\psi(x)$ equals $\theta$.

Second, and with similar proofs, if $\phi$ fixes some geodesic line $L$ then $\psi \phi \psi^{-1}$ also fixes a line, namely $\psi(L)$. Furthermore, if $\phi$ translates a distance $d$ along $L$ then $\psi \phi \psi^{-1}$ translates a distance $d$ along $\psi(L)$.

Third, a parabolic transformation is one which fixes a single point on the circle at infinity and fixes no finite point.

Here's where things get interesting. Now you have to convince yourself of two further issues.

1. Every o.p. isometry either fixes a point and rotates about it by some angle, or fixes some geodesic line and translates along that line by some distance, or is a parabolic isometry, or is the identity.
2. Any two o.p. isometries which are rotations by the same angle, or are translations by the same distance, are conjugate. A similar statement holds for parabolic isometries, which I'll hold off until the end.

It is possible to prove this synthetically, but Möbius transformations certainly help. Here's an outline.

As usual we represent Möbius transformations by elements of $PSL(2,\mathbb{R}) = SL(2,\mathbb{R}) / \pm I$. Here's some algebra facts which you can easily verify.

• Given $M \in SL(2,\mathbb{R})$, $\text{trace}(M)$ is a conjugacy invariant in $SL(2,\mathbb{R})$. Also, its absolute trace $|\text{trace}(M)|$ is a conjugacy invariant in $PSL(2,\mathbb{R})$.
• If $|\text{trace}(M)|$ is not equal to 2 then it is a complete conjugacy invariant, i.e. if $M_1,M_2$ have the same absolute trace not equal to 2 then they are conjugate in $PSL(2,\mathbb{R})$. The case of absolute trace equal to $2$ corresponds to parabolic transformations and is not quite as clean; I'll skip that until the end.

Now, blend the geometry and the algebra: for each value of the absolute trace that is not equal to $2$, construct an example, verify that it is one of the isometries in the above list, and verify that every distinct isometry in the above list corresponds to exactly one value of the absolute trace. Having done that, the classification is complete! (Outside of the issue that I have ignored the parabolic case where absolute trace equals 2).

For example, absolute trace equal to zero is represented by $\pmatrix{0 & 1 \\ -1 & 0} : z \mapsto -\frac{1}{z}$ which is a rotation about the fixed point $0+1i$ in the upper half plane, with rotation angle $\pi$. More generally, absolute trace in the interval $[0,2)$ is always a rotation about a fixed point, with angles varying over $(0,\pi]$ by a simple strictly monotonic continuous function $[0,2) \mapsto (0,\pi]$ (whose formula some people remember but I never can).

For another example, absolute trace equal to $\frac{5}{2}$ is represented by $\pmatrix{2 & 0 \\ 0 & 1/2}$ which is a translation along the line $\text{RealPart}(z)=0$ by a distance $\ln(4)$. More generally, absolute trace in the inverval $(2,\infty)$ is always a translation along a line, with translation distance varying over $(0,\infty)$ by a simple strictly monotonic continuous function $(2,\infty) \mapsto (0,\infty)$ (again, with an explicit formula).

A few last words about absolute trace equal to $2$. Each of these represents either the identity or an o.p. parabolic isometry. In the full group of generalized Möbius transformations there is just one conjugacy class of parabolic isometry. But in $\text{PSL}(2,\mathbb{R})$ there are two conjugacy classes, one rotating "positively" around the fixed point at infinity and the other rotating "negatively".

In my PhD. thesis I'm classifying hyperbolic isometries in The Poincaré disk model based on their generalized Möbius transformation matrices in section 2.2.4. My target categories are translations, rotations, limit rotations, reflections and glide reflections.

Firstly I'd distinguish Möbius transformations from anti-Möbius transformations. As an alternative one could avoid dealing with anti-Möbius transformations and instead consider equivalence classes of points related by inversion in the unit circle, or use the half plane model and consider the lower half plane to be a mirror image of the upper half plane. In that case the distinction would be for Möbius transformations which exchange the two mirror images, as these would be the orientation-reversing ones.

Secondly, I'd look at fixed points. A translation has two ideal fixed points, while a rotation has a finite fixed point and its mirror image after reflection in the unit circle (resp. the real line for the half plane model). The former is usually called a hyperbolic Möbius transformation, the latter an elliptic one. In the limit between these two situations, there is a single ideal fixed point with multiplicity two, and the operation is a limit rotation which fixes horocycles through the fixed ideal point. This class of Möbius transformations is called parabolic.

For ani-Möbius transformations, there may be a whole geodesic of fixed points, in which case one has a reflection. Otherwise there will be two ideal fixed points denoting the endpoints of the axis of glide reflection.

Note that the Euclidean concept of a “translation by a vector” doesn't make sense in hyperbolic geometry. Every hyperbolic translation has a designated axis of translation which is fixed by that operation, but since there is no family of parallel lines in the Euclidean sense, you don't have parallel lines fixed as well.