Subgroup of $\mathrm{GL}_n$ stabilizing linear subspace skew-symmetric matrices

Here is an outline of the argument that shows that the $\mathrm{SL}_6(\mathbb{C})$-stabilizer of the generic $3$-plane $W\subset\Lambda^2(\mathbb{C}^6)$ has dimension $1$, not $0$, as (apparently) claimed.

First, note that the cone $C_2\subset \Lambda^2(\mathbb{C}^6)$ consisting of the elements $b\in \Lambda^2(\mathbb{C}^6)$ that satisfy $b^3 = 0$ is a hypersurface of degree $3$ in $\Lambda^2(\mathbb{C}^6)\simeq\mathbb{C}^{15}$. It is irreducible but not smooth, as it is singular along the locus $C_1\subset C_2$ consisting of the elements that satisfy $b^2=0$, and $C_1$ itself is a smooth cone of dimension $9$ (i.e., its only singularity is the origin $b=0$).

Thus, a generic $3$-plane $W$ in $\Lambda^2(\mathbb{C}^6)$ will only meet $C_1$ at the origin and will not be tangent to $C_2$ anywhere. Thus, the intersection $W\cap C_2$ will be a smooth $2$-dimensional cone that projectivizes to a smooth cubic curve in $\mathbb{P}W\simeq \mathbb{P}^2$.

The group $G\subset \mathrm{SL}_6(\mathbb{C})$ that stabilizes $W$ must act on $\mathbb{P}W$ as symmetries of the nonsingular cubic curve and hence must act as a finite group on $\mathbb{P}W$. By passing to a subgroup $G'\subset G$ of finite index in $W$, we can assume that $G'$ acts trivially on $\mathbb{P}W$ and hence as a scalar multiple of the identity on $W$. However, if we let $b\in W$ be an element that satisfies $b^3\not=0$, then $G'$ must preserve $b^3$ and hence, it can at most multiply $b$ by a nontrivial cube root of unity. Again passing to a subgroup $G''\subset G'$ of index at most $3$, we can arrange that $G''$ acts trivially on $W$.

Let $P\subset W$ be a plane that intersects $W\cap C_2$ in three distinct lines. This means that $P$ has a basis consisting of two elements $b_1\not\in C_2$ and $b_2$ such that the polynomial $p(\lambda)$ that satisfies $$ (b_2-\lambda b_1)^3 = p(\lambda) b_1^3 $$ has $3$ distinct roots. After a change of basis in $P$, I can assume that those roots are $1$, $2$, and $3$.

Then, by an elementary argument, there is a basis $e_1,\ldots,e_6$ of $\mathbb{C}^6$ such that $$ b_1 = e_1\wedge e_2 + e_3\wedge e_4 + e_5\wedge e_6 \quad\text{and}\quad b_2 = e_1\wedge e_2 + 2e_3\wedge e_4 + 3e_5\wedge e_6 $$ Moreover, the fact that $G''$ fixes $b_1$ and $b_2$ implies that it also fixes the individual monomial terms $e_1\wedge e_2$, $e_3\wedge e_4$, and $e_5\wedge e_6$.

Let $Q_i\subset\mathbb{C}^6$ be the $2$-plane spanned by $e_{2i-1},e_{2i}$. Then $G''\subset \mathrm{SL}(Q_1) \times\mathrm{SL}(Q_2)\times \mathrm{SL}(Q_3)$, and so, of course, $Q_i\simeq Q_i^*$ as $G$-modules. Now let $b_3\in W$ be linearly independent from $b_1$ and $b_2$. Using the $G''$-module decomposition $$ \begin{aligned} \Lambda^2(\mathbb{C}^6) &= \Lambda^2(Q_1\oplus Q_2\oplus Q_3) \\ &= \Lambda^2(Q_1)\oplus\Lambda^2(Q_2)\oplus\Lambda^2(Q_3)\oplus Q_1{\otimes}Q_2\oplus Q_2{\otimes}Q_3\oplus Q_3{\otimes}Q_1\\ &\simeq\mathbb{C}\oplus\mathbb{C}\oplus\mathbb{C}\oplus Q_1{\otimes}Q_2^*\oplus Q_2{\otimes}Q_3^*\oplus Q_3{\otimes}Q_1^*, \end{aligned} $$ one can decompose $b_3$ into a sum of the three basic monomials that occur in $b_1$ and $b_2$ plus a triple of linear maps $L_i:Q_{i+1}\to Q_i$. For generic $W$ with basis $b_1$, $b_2$, and $b_3$ as above, these $L_i$ will be isomorphisms, and they will have to be $G''$-module isomorphisms in order for $b_3$ to be fixed by $G''$, whose elements can be thought of as triples of elements $(g_1,g_2,g_3)$ where $g_i\in\mathrm{SL}(Q_i)$. However, the relations $L_ig_{i+1}=g_iL_i$ (necessary for $G''$ to fix $b_3$) will then determine $g_2$ and $g_3$ in terms of $g_1$ and the $L_i$. Moreover, $L = L_1L_2L_3:Q_1\to Q_1$ must commute with $g_1$. Conversely, if $g_1$ commutes with $L$ then it determines an element of $G''$. However, $L$ will always have a positive dimensional commutator in $\mathrm{SL}(Q_1)$ (generically of dimension $1$), so $G''$ always has dimension at least $1$.

In the following, a linear group is a (closed) subgroup $G$ of some $GL(V)$. Now the stabilizer of a generic $d$-space in $V$ is the same as the stabilizer in $G\times GL(d)$ of a generic vector of $V\otimes\mathbb C^d$. Thus, in our case we have to investigate the generic stabilizer of $GL(n)\times GL(d)$ acting on $\wedge^2\mathbb C^n\otimes\mathbb C^ d$ with $1\le d\le n(n-1)/4$ and determine when it is infinite.

Determining the generic isotropy group $H$ for a linear action of a reductive group $G$ is a classical problem of invariant theory. It was pretty much settled by the Vinberg school approx. 50 years ago. More specifically: In

Andreev, E. M.; Vinberg, È. B.; Èlašvili, A. G.: Orbits of highest dimension of semisimple linear Lie groups. (Russian) Funkcional. Anal. i Priložen. 1 1967 no. 4, 3–7

the authors derive a numerical criterion for $H$ to be infinite (the first theorem of that paper). The argument is really ingeneous and I recommend reading the paper. From their criterion they derive the classification of irreducible simple linear groups with infinite $H$. The table became later notorious because the very same table showed up in various different classification projects (see, e.g., the appendix to Mumford-Fogarty(-Kirwan)).

Probably, the numerical criterion alone is already enough to settle the finiteness of $H$ in the case at hand. But that is not necessary. In the follow-up paper

Èlašvili, A. G. Stationary subalgebras of points of general position for irreducible linear Lie groups. (Russian) Funkcional. Anal. i Priložen. 6 (1972), no. 2, 65–78

Èlašvili (almost, see below) settles the case of arbitrary irreducible linear groups. The answer for non-simple groups is given in Tables 5 and 6. The cases $\wedge^2\mathbb C^n\otimes\mathbb C^ d$ with infinite $H$ are $(n,d)=(n,1), (n,2), (4,3), (5,3)$, and $(6,3)$. So $d\ge4$ does not occur, as claimed.

There is more to say. Sifting through the tables one will notece that Robert Bryant's case $(n,d)=(6,3)$ is missing in Table 5. This case along with three more missing cases was pointed out in

Popov, A. M. Irreducible semisimple linear Lie groups with finite stationary subgroups of general position. (Russian) Funktsional. Anal. i Prilozhen. 12 (1978), no. 2, 91–92.

See the second paragraph. As for reliability of these results, the whole classification was recovered in

Knop, Friedrich; Littelmann, Peter Der Grad erzeugender Funktionen von Invariantenringen. (German) [The degree of generating functions of rings of invariants] Math. Z. 196 (1987), no. 2, 211–229

There we dealt with a slightly broader classification problem where we nevertheless had to do all calculations from scratch. So we were not using Èlašvili's tables directly. That's why I am pretty positive that the tables are complete.

Here is the worst possible proof of all but one case, namely $m=r=3$ (but this is not included in your question as stated, though I think it is claimed in the paper.)

Let $r$ denote the dimension of a generic subspace $W$ of $\bigwedge^2\mathbb{C}^{2m}$. We assume $3\leq r\leq\frac{1}{2}\operatorname{dim}(\bigwedge^2\mathbb{C}^{2m})$ (Voisin assumes a possibly weaker upper bound, but this is OK by duality.) Our goal is to show that no nontrivial element of $\operatorname{PGL}_{2m}$ stabilizies $W$.

The first observation we make is that if $g$ stabilizes $W$, then the semisimple and unipotent parts $g_{ss},g_{un}$ also stabilize $W$ (because they are polynomials in $g$). If $g_{ss}$ is nontrivial, we can take some polynomial in it to get a semisimple element with exactly two eigenvalues. If $g_{un}$ is nontrivial, we can take some shifted power $(g_{un}-1)^k+1$ which is nontrivial and only has Jordan blocks of length $1$ and $2$. In conclusion: it suffices to show that $W$ is not stabilized by any semisimple element with two eigenvalues nor any unipotent element with Jordan blocks of length $\leq 2$. We treat these two cases separately.

In both cases, we will do an incidence correspondence argument. Namely, we look at the variety of pairs $(g,W)$, with $g$ nontrivial and semisimple w/ two eigenvalues (or unipotent w/ $(g-1)^2=0$) and show this variety has dimension less than the dimension of $\operatorname{Gr}(r,m(2m-1)).$ We do this by splitting this variety into many many (but finitely many) strata, and showing the dimension bound for each strata.

The semisimple case. Our semisimple element $g$ with two eigenvalues gives a partition $d+e=2m$, given by the dimensions of each eigenspace. The action of $g$ on $\bigwedge^2\mathbb{C}^{2m}$ decomposes it as a sum of eigenspaces $\mathbb{C}^{\binom{d}{2}}\oplus\mathbb{C}^{de}\oplus\mathbb{C}^{\binom{e}{2}}.$

A subspace $W$ is stabilized by $g$ iff it is a direct sum of subspaces in each component. Letting the dimension of these subspaces be given by $f_1,f_2,f_3$, we see that we have $f_1(\binom{d}{2}-f_1)+f_2(de-f_2)+f_3(\binom{e}{2}-f_3)$ degrees of freedom to choose $W$.

Consider the strata given by fixing d,e, and the $f_i$. We have a $2de+1$-dimensional space of choices of $g$; $de$ for a subspace of dimension $d$, another $de$ for a subspace of dimension $e$, and a final degree of freedom for the ratio between the two eigenvalues. So our desired inequality becomes

$$2de+1+f_1(\binom{d}{2}-f_1)+f_2(de-f_2)+f_3(\binom{e}{2}-f_3)<r(m(2m-1)-r).$$

Assume this inequality is false. Consider the choice of $f_i$ that maximizes the left hand side. As $f_1+f_2+f_3=r\leq\frac{1}{2}m(2m-1),$ we must have $f_1\leq\frac{1}{2}\binom{d}{2}$, $f_2\leq\frac{1}{2}de$, $f_3\leq\frac{1}{2}\binom{e}{2}$ (by the maximizing assumption). Now rewrite (the negation of) the inequality as:

$$2de+1\geq (f_2+f_3)(\binom{d}{2}-f_1)+(f_1+f_3)(de-f_2)+(f_1+f_2)(\binom{e}{2}-f_3).$$

Assume WLOG that $d\geq e$. As $f_2\leq\frac{1}{2}de$, we must have $f_1+f_3\leq 4.$ The right hand side contains $f_2(\binom{d}{2}+\binom{e}{2}-f_1-f_3)$ as a summand, and because $m\geq 3$, we have $\binom{d}{2}+\binom{e}{2}\geq\frac{2}{3}de,$ so $\binom{d}{2}+\binom{e}{2}-f_1-f_3\geq\frac{1}{3}de,$ so $f_2\leq 6$.

We see in particular that $2de+1\geq f_2(\binom{d}{2}+\binom{e}{2}-4)+(f_1+f_3)(de-6),$ and since $(f_2+f_3)+(f_1+f_3)\geq r\geq 3$, either $\binom{d}{2}+\binom{e}{2}-4$ or $de-6$ is $\leq\frac{2de+1}{3}$. If $de-6\leq\frac{2de+1}{3}$, then $de\leq 19$ and $m \leq 10$. while if $\frac{2de+1}{3}\geq\binom{d}{2}+\binom{e}{2}-4,$ then we have $\frac{2m^2+1}{3}\geq\frac{2de+1}{3}\geq\binom{d}{2}+\binom{e}{2}-4=\frac{d^2+e^2}{2}-m-4\geq m^2-m-4,$ so $m\leq 5.$

In summary, we have $m\leq 10.$ Using a computer to check all such cases (as I said, this is the worst possible proof), we see that the only possible case is $f_1=0,f_2=3,f_3=0,d=e=m=3$, but this is exactly the case I excluded at the start.

Now let do the same argument for unipotent $g$ with $(g-1)^2=0$. Let $n$ denote the number of nontrivial Jordan blocks, or equivalently, the dimension of $\operatorname{im}(g-1).$ Then the action of $g$ on $\bigwedge^2\mathbb{C}^{2m}$ has $n(2m-n-1)$ nontrivial blocks (each of length $2$).

The data of a preserved subspace $W\subseteq\bigwedge^2\mathbb{C}^{2m}$ is equivalent to the data of a subspace $W'\cong W\cap\operatorname{im}(g-1)\subseteq\operatorname{im}(g-1)$ and a subspace $W/W'\subseteq (g-1)^{-1}W'/W'.$ If the dimension of $W'$ is given by $d$, then there are $d(n(2m-n-1)-d)+(r-d)(m(2m-1)+d-n(2m-n-1)-r)$ degrees of freedom. Meanwhile, the data of a choice of $g$ is equivalent to the choice of the subspace $\operatorname{im}(g-1)$ and of a map $\mathbb{C}^{2m}/\operatorname{im}(g-1)\rightarrow \operatorname{im}(g-1)$, which gives $n(2m-n)+n(2m-n)=2n(2m-n)$ degrees of freedom. So the inequality we want is

$$d(n(2m-n-1)-d)+(r-d)(m(2m-1)+d-n(2m-n-1)-r)+2n(2m-n)< r(m(2m-1)-r).$$

We can rearrange this to

$$2d^2+(m(2m-1)-2r-2n(2m-n-1))d-2n(2m-n)+rn(2m-n-1) > 0.$$

Assume otherwise. Note that $\frac{2m-n-1}{2m-n}\geq\frac{2}{3}$, so $-2n(2m-n)+rn(2m-n-1)\geq 0,$ with equality only when $m=n=r=3$ (and so we land in the excluded case again.) So we must have $m(2m-1)-2r-2n(2m-n-1)<0$ and, by the discriminant test,

$$8(rm(2m-n-1)-2n(2m-n))\leq(m(2m-1)-2r-2n(2m-n-1))^2.$$

Note that we have $m(2m-1)-2r-2n(2m-n-1)\geq m-2r$ and $m(2m-1)-2r-2n(2m-n-1)\geq -2n(2m-n-1)$. Applying these inequalities, we see that

$$8(rm(2m-n-1)-2n(2m-n))\leq(2r-m)2n(2m-n-1)$$.

Again applying $2(2m-n)\leq 3(2m-n-1),$ we get

$$8(rm-3n)(2m-n-1)\leq(2r-m)2n(2m-n-1)$$

or

$$8(rm-3n)\leq(2r-m)2n.$$

This implies

$$8(rn-3n)\leq(2r-m)2n$$

and

$$8r-24\leq 4r-2m$$

$$4r\leq 24-2m$$

so $m\leq 6$. Again, we check all these cases by computer. All of them either have $m=r=3$ or have equality in the dimension bound. But in this case, equality suffices, because all subgroups of $G$ containing a unipotent element have positive dimension.