Subgroups of $SL_2(\mathbb R)$ which contain $SL_2(\mathbb Z)$ as a finite index subgroup

$G = SL_2(\mathbb{Z})$ is indeed the only possibility. This follows from the structure of the quotient orbifold $\mathcal{O} = \mathbb{H}^2 / SL_2(\mathbb{Z})$, which is a once-punctured sphere with a $\mathbb{Z}/2\mathbb{Z}$ cone point and a $\mathbb{Z}/3\mathbb{Z}$ cone point. The key fact is that the orbifold $\mathcal{O}$ is not a proper orbifold-covering space of any other oriented 2-orbifold. The proof of this fact is essentially an Euler characteristic calculation: given that $$\chi(\mathcal{O}) = 2 - 1 - (1 - \frac{1}{2}) - (1 - \frac{1}{3}) = -\frac{1}{6} $$ one then verifies that no fraction of the form $-\frac{1}{6n}$ with $n \ge 2$ can be the Euler characteristic of any finite type oriented 2-orbifold having at least one puncture.


Even for $Sp_{2g}$ the only possibility is $Sp_{2g}(\mathbb{Z})$. To see this, suppose $\Gamma \subset Sp_{2g}(\mathbb{R})$ is a subgroup containing $Sp_{2g}(\mathbb{Z})$ as a finite index subgroup. Suppose we prove that $\Gamma $ consists of rational symplectic matrices. The fact that $Sp_{2g}(\mathbb{Z}_p)$ is a $maximal$ compact subgroup of $Sp_{2g}(\mathbb{Q}_p)$ for all primes $p$ (and strong approximation) implies that $Sp_{2g}(\mathbb{Z})$ is a maximal arithmetic subgroup of $Sp_{2g}(\mathbb{Q})$ (edited: if $\Gamma \subset Sp_{2g}(\mathbb{Q})$ contains $Sp_{2g}(\mathbb{Z})$ as a finite index subgroup, then $\Gamma $ is the integral symplectic group ) and hence this implies that $\Gamma =Sp_{2g}(\mathbb{Z})$.

Thus it remains to prove that $\Gamma $ consists of rational matrices. This is essentially a Galois cohomology argument. Since $\Gamma$ contains $Sp_{2g}(\mathbb{Z})$ as a finite index subgroup, in particular, there is a finite index subgroup $N\subset Sp_{2g}(\mathbb{Z})$ $normalised$ by $\Gamma$. $N$ acts irreducibly on $\mathbb{C}^{2g}$ (for example, snce $N$ is Zariski dense in the complex symplectic group). Thus, by Burnside's theorem, there is a basis $e_1,\cdots,e_N$ of $2g\times 2g$ rational matrices consisting of elements of $N$.

Suppose $x\in \Gamma \subset Sp_{2g}(\mathbb{R})\subset Sp_{2g}(\mathbb{C})$. Under conjugation by $x$, the basis $e_i$ goes into elements of $N \subset Sp_{2g}(\mathbb{Z})$ and hence $xe_ix^{-1}$ is a rational linear combination of the matrices $e_i$. If $\sigma \in Aut(\mathbb{C})$ then $\sigma (x)e_i\sigma (x)^{-1}=xe_ix^{-1}$. The irreducibility of $N$ now implies that the cocycle $x^{-1}\sigma (x)$ is a scalar matrix $\lambda$. Since $x$ preserves the symplectic form, this implies that $\lambda ^2=1$, and hence $\lambda =\pm 1$. If $\lambda =1$ for all $\sigma$ then $x$ is rational, and that is what we set out to prove.

If the Galois cocycle $\lambda =\lambda _{\sigma}= -1$, then we may write $\lambda =i^{-1}\sigma (i)$ (where $i^2=-1$) and hence $h=i^{-1}x$ is a rational matrix: $h^{-1}\sigma (h)=1$ for all $\sigma$. But, $\Gamma $ consists of real matrices, and hence $x$ is a real matrix, and hence, the equation $x=ih$ for some rational matrix $h$ is impossible. Thus, $\lambda =-1$ is ruled out.

So essentially, the fact that over the $p$-adic field $\mathbb{Q}_p$, the group $Sp_{2g}(\mathbb{Z}_p)$ is maximal compact implies that $Sp_{2g}(\mathbb{Z})$ is a subgroup of $Sp_{2g}(\mathbb{R}$ such that every intermediate subgroup contains $Sp_{2g}(\mathbb{Z})$ as an infinite index subgroup.


I answer here the case of $\mathrm{SL}_2$, with actually an argument carrying over $\mathrm{SL}_n$. Namely, $\mathrm{SL}_n(\mathbf{Z})$ is maximal among lattices in $\mathrm{SL}_n(\mathbf{R})$. Indeed, let $\Lambda$ be an overgroup of finite index.

First assume that $\Lambda\subset\mathrm{SL}_n(\mathbf{Q})$. Then there is a bound on denominators in $\Lambda\subset\mathrm{M}_n(\mathbf{Q})$. It follows that the subgroup of $\mathbf{Q}^n$ generated by the $g\mathbf{Z}^n$ when $g$ ranges over $\Lambda$, is a lattice. So $\Lambda$ preserves this lattice and hence is conjugate to a subgroup of $\mathrm{SL}_n(\mathbf{Z})$. But since the automorphism group of $\mathrm{SL}$ preserves the volume, this implies that $\Lambda=\mathrm{SL}_n(\mathbf{Z})$.

Now we have to show that $\Gamma$ is contained in $\mathrm{SL}_n(\mathbf{Q})$. To show this, let us first understand orbits in $\mathbf{R}^n$ of finite index subgroups $\Xi$ of $\mathrm{SL}_n(\mathbf{Z})$. Start from a nonzero vector $v$, we wish to describe the closure $X$ of the additive subgroup generated by the orbit $\Lambda v$. Note that $X$ is the closure of $Av$, where $A$ is the $\mathbf{Z}$-subalgebra generated by $\Gamma$. For some $m\ge 1$, all elementary matrices $e_{ij}(m)=I+E_{ij}(m)$ belong to $\Xi$, and hence $E_{ij}(m)$ belongs to $A$. So, denoting by $(e_i)$ the canonical basis, $mv_je_i$ belongs to $X$ for all $i\neq j$. If the $v_j$ and $v\in X$ for $j\neq i$ generate a dense subgroup of $\mathbf{R}$, we deduce that the line $\mathbf{R}e_i$ is contained in $X$, and reiterating, we see that $\mathbf{R}e_j$ is contained in $X$ for all $j$ and hence that $X=\mathbf{R}$. We can do this as soon as $n\ge 3$ and $v$ is not scalar multiple of a rational vector. If $n=2$, assuming that $v$ is not multiple of a scalar vector, we can assume $v=(1,t)$ with $t$ irrational. Then $X$ contains the lattice $t\mathbf{Z}\times\mathbf{Z}$. Clearly $v$ has infinite order modulo this lattice, so the closure of $\mathbf{Z}v$ modulo this lattice is non-discrete, hence $X$ is not discrete, thus contains a line. Applying either $e_{12}(m)$ or $e_{21}(m)$ to this line yields another line and hence $X=\mathbf{R}^2$.

To summarize, we have shown that for every $v$ that is not scalar multiple of a rational vector, the subgroup generated by $\Xi v$ is dense. Thus, the lattices preserved by $\Xi$ are all scalar multiples of rational lattices (i.e., contained in $\mathbf{Q}^n$).

If $g\in\Lambda$, there exist two finite index subgroups $\Xi_1$ and $\Xi_2$ of $\mathrm{SL}_n(\mathbf{Z})$ such that $g\Xi_1=\Xi_2$. From the previous fact, we deduce that $g$ maps rational lattices to scalar multiples of rational lattices. Thus we can write $g=\lambda g'$ with $g'$ a rational matrix. The determinant condition implies that $1=\lambda^n\det(g)$, and thus $\lambda^n$ is rational. Also, $\lambda$ is unique in $\mathbf{R}^*/\mathbf{Q}^*$ and $g\mapsto \lambda=\lambda_g$ is a homomorphism into this group. The kernel is the intersection $\Lambda\cap\mathrm{SL}_n(\mathbf{Q})$, which has been shown to be $\mathrm{SL}_n(\mathbf{Z})$. So $\mathrm{SL}_n(\mathbf{Z})$ is normal in $\Lambda$.

So now we need to understand the normalizer of $\mathrm{SL}_n(\mathbf{Z})$. Using that $\mathrm{SL}_n(\mathbf{Z})$ is transitive on primitive elements in $\mathbf{Z}^n$, we see that the lattices preserved by $\mathrm{SL}_n(\mathbf{Z})$ are precisely the scalar multiples of $\mathrm{Z}^n$. So if an element normalizes it and preserves the volume, it has to preserve $\mathrm{Z}^n$. This shows that $\mathrm{SL}_n(\mathbf{Z})$ is self-normalized in $\mathrm{SL}_n(\mathbf{R})$, and finally $\Lambda=\mathrm{SL}_n(\mathbf{Z})$.

A similar argument shows that every lattice containing a finite index subgroup of $\mathrm{SL}_n(\mathbf{Z})$ is actually contained in a conjugate of $\mathrm{SL}_n(\mathbf{Z})$ by some rational matrix.