Is there a scheme parametrizing the closed subgroups of an algebraic group?

If $k$ is a field, the Hilbert functor of closed subschemes of a $k$-scheme of finite type $X$ is representable if $X$ is projective (or more generally quasiprojective if you restrict to projective subschemes), but not in general.

Let me give a counterexample for subgroups, taking $G=\mathbb{G}_a\times\mathbb{G}_m$ and $k$ of characteristic zero. So, let $H$ be the functor from $(k\text{-Schemes})^\circ$ to (Sets) given by $$H(T):=\{\text{closed $T$-subgroup schemes of $G\times T$, flat over $T$}\}.$$ For $n\in\mathbb{N}$, put $T_n:=\mathrm{Spec}\,k[t]/(t^{n+1})$ and $T_\infty=\mathrm{Spec}\,k[[t]]$. Observe that if $H$ were representable we would have $H(T_\infty)=\varprojlim_n H(T_n)$. Define $\Gamma_n\in H(T_n)$ as the "graph of the exponential": $$\Gamma_n:=\mathrm{Spec}\,\left(k[t,x,y,y^{-1}]\,/\,(t^{n+1}, \,y-\exp\,(tx))\right)$$ which makes sense because $t^{n+1}=0$, so the exponential is actually a polynomial. Clearly $(\Gamma_n)_{n\in\mathbb{N}}$ is an element of $\varprojlim_n H(T_n)$, but there is no subscheme of $G\times T_\infty$ inducing $\Gamma_n$ on $G\times T_n$ for all $n$.

Of course, there are representability results if you restrict to certain types of subgroups, e.g. Borel subgroups, or subgroups of multiplicative type (see e.g. SGA 3, exposé XI).

Here is an affirmative answer in the sense of algebraic spaces under a reductivity hypothesis on the subgroups, using some hard input from SGA3. (The representability by a scheme for the functor classifying parabolic subgroups is well-documented in the literature in various settings; see section 3 of Exp. XXVI in SGA3 for the case of a general base scheme.) Let $G \rightarrow S$ be a smooth affine group scheme, and let $F$ be the functor on $S$-schemes $$F(T) = \{H \subset G_T \mbox{ a reductive closed subgroup scheme}\}$$ where "reductive $T$-group" means "smooth affine $T$-group with connected reductive geometric fibers".

For any $T$ and $H \in F(T)$, the isomorphism type of the fiber of $H \rightarrow T$ over a geometric point $t$ of $T$ is determined by the root datum for $H_t$, and it is a result in the relative theory of reductive groups that this root datum is "locally constant" in $t$. More precisely, if $\Sigma$ is the set of isomorphism classes of root data and for $\sigma \in \Sigma$ we define $T_{\sigma}$ to be the set of points of $T$ over which the geometric fiber of $H$ has root datum isomorphic to $\sigma$ then the $T_{\sigma}$'s are pairwise disjoint and open in $T$. Thus, if we define the subfunctor $F_{\sigma} \subset F$ to consist of those "points" of $F$ whose associated geometric fibers $H_t$ all have root datum $\sigma$ then the $F_{\sigma}$'s are open subfunctors which cover $F$ disjointly. More specifically, the representability of $F$ is equivalent to that of the $F_{\sigma}$'s, and in particular, if $F_{\sigma}$ is represented by some $M_{\sigma}$ then $F$ is represented by the disjoint union of the $M_{\sigma}$'s. This is the same as the game whereby the construction of Hilbert schemes is reduced to the case of a fixed Hilbert polynomial.

Now we fix $\sigma$ and focus on $F_{\sigma}$. Consider the associated Chevalley group $G_{\sigma}$ over $\mathbf{Z}$ (i.e., the unique "split" reductive $\mathbf{Z}$-group whose geometric fibers have root datum $\sigma$). By a hard fact in the general theory of reductive group schemes, a reductive group $H \rightarrow T$ has all geometric fibers with root datum $\sigma$ if and only if $H$ and $(G_{\sigma})_T$ become isomorphic over an etale cover of $T$. So, informally, $F_{\sigma}$ is the functor of closed subgroup schemes which are etale forms of $(G_{\sigma})_T$. This motivates us to look at the smooth separated Hom-scheme $$\mathscr{H} := \underline{\rm{Hom}}((G_{\sigma})_S,G)$$ whose set of $T$-points is the set of $T$-group scheme homomorphisms $(G_{\sigma})_T \rightarrow G_T$ (see SGA3 XXIV Cor. 7.2.3 for existence) naturally in $T$. Over $\mathscr{H}$ there is a "universal homomorphism" $$f:(G_{\sigma})_{\mathscr{H}} \rightarrow G_{\mathscr{H}}.$$

I claim that the condition of being a closed immersion is represented by an open subscheme $X$ of $\mathscr{H}$, so then $X$ represents the functor assigning to any $S$-scheme $T$ the set of reductive closed $T$-subgroup schemes $H \subseteq G_T$ equipped with a $T$-group isomorphism $\varphi:H \simeq (G_{\sigma})_T$. Let's grant this. There is an evident map $X \rightarrow F_{\sigma}$ that forgets $\varphi$, and this is a surjection for the etale topology since all $H \in F_{\sigma}(T)$ become isomorphic to $(G_{\sigma})_T$ over an etale cover of $T$ (as noted above). Thus, $X \rightarrow F_{\sigma}$ is a torsor for the etale topology relative to the natural action of the automorphism scheme of $G_{\sigma}$. That is, $F_{\sigma}$ is the quotient sheaf for $X$ modulo the faithful action of the smooth separated $S$-group $({\rm{Aut}}_{G_{\sigma}/\mathbf{Z}})_S$ that is an extension of the constant etale $S$-group ${\rm{Aut}}(\sigma)_S$ by the adjoint semisimple $S$-group $(G_{\sigma}^{\rm{ad}})_S := (G_{\sigma}/Z_{G_{\sigma}})_S$, so by Artin's work on algebraic spaces it follows that this quotient sheaf $F_{\sigma}$ is an algebraic space, even smooth (with the algebraic space $X/(G_{\sigma}^{\rm{ad}})_S$ as an $S$-smooth etale cover).

If the root datum $\sigma$ is semisimple then $(G_{\sigma})_S$ is semisimple and so its the automorphism scheme is $S$-affine, hence quasi-compact over $S$, so $F_{\sigma}$ is quasi-separated over $S$. Thus, in such cases the valuative criterion for separatedness is applicable, giving that $F_{\sigma}$ is also separated. For general $\sigma$ separatedness might still hold but one would have to look more closely at the contributions from the derived group and maximal central torus of $G_{\sigma}$; probably nobody cares, so I won't delve into that.

It remains to build the open subscheme $X$ as above, so we show more generally:

Theorem. Let $f:G' \rightarrow G$ be a homomorphism between affine fppf group schemes over a scheme $S$, with $G'$ reductive. Then the set $U$ of $s \in S$ such that $f_s$ is a closed immersion is open in $S$ and $f_U:G'_U \rightarrow G_U$ is a closed immersion.

Maybe this is handled somewhere in SGA3; I don't remember offhand. The proof of this turned out to be a bit more involved than I expected, but I may be overlooking something simpler.

Proof: We may and do assume $S$ is noetherian. By the very difficult SGA3, XVI, 1.5(a), $f$ is a closed immersion if and only if it is a monomorphism. Hence, once we show that $U$ is open it will suffice to show that the $U$-group $\ker(f_U)$ is trivial. The $S$-group $\ker(f)$ is relatively affine and finite type over $S$, so it is trivial if and only if all of its fibers are trivial (by Nakayama's Lemma), so it remains to prove that $U$ is open.

By "spreading out" considerations and passing to local rings on $S$ at such points $s$, we may assume $S$ is local noetherian and just need to check that $\ker f = 1$ if $\ker(f_s) = 1$ for the closed point $s$ of $S$. Since it is enough to check fiberwise triviality, by picking a dvr whose special point dominates $s$ and whose generic point dominates another chosen point of $S$ we may assume $S = {\rm{Spec}}(A)$ for a dvr $A$, say with fraction field $K$. Thus, the schematic closure of $f_K(G'_K)$ inside $G$ is a flat affine $A$-group of finite type that we shall call $\mathbf{G}$. Clearly $f$ factors through an $A$-homomorphism $G' \rightarrow \mathbf{G}$ that is a closed immersion between special fibers and surjective between generic fibers. Thus, comparison of fiber dimensions implies that $f$ has finite kernel between generic fibers, so $\ker f$ is quasi-finite over $S$.

The generic fiber $\mathbf{G}_K = f_K(G'_K)$ is a quotient of $G'_K$ and so is $K$-smooth. Thus, we can form the group smoothening $\mathbf{G}' \rightarrow \mathbf{G}$ (see Theorem 5 in 7.1 of "Neron Models"); by construction this is a smooth $A$-group with the same $K$-fiber as $\mathbf{G}$ and it is moreover affine since $\mathbf{G}$ is affine. By its universal property, we get a factorization $G' \rightarrow \mathbf{G}'$ whose special fiber has trivial kernel and hence is a closed immersion. It suffices to show that this is a closed immersion between $K$-fibers, so we may replace $G$ with $\mathbf{G}'$ to reduce to the case of a smooth target $G$ with $f_K$ is surjective (so $G_K$ is connected). Since $f$ induces an isogeny between the smooth identity components on fibers over $S$, the identity components of the fibers of $G'$ are reductive.

Let $G^0$ be the open complement of the non-identity component of the special fiber, so $G^0 \rightarrow S$ is smooth, separated, and finite type with connected reductive fibers. By a clever translation argument in SGA3, XXV, section 4, a quasi-finite homomorphism between separated flat group schemes of finite type over a Dedekind base is an affine morphism, so $G^0$ is affine (as $G$ is). Thus, $G^0$ is a reductive group scheme, so the open immersion $j:G^0 \hookrightarrow G$ between smooth affine $A$-groups is also a closed immersion (SGA3, XVI, 1.5(a)). The defining ideal for this closed immersion vanishes on the $K$-fibers and hence vanishes; i.e., $G^0 = G$. We conclude that $f:G' \rightarrow G$ is a fiberwise isogeny for dimension, quasi-finiteness, and connectedness reasons. But it is a general (hard!) fact that a fiberwise isogeny between reductive group schemes is always finite flat. Since $f$ is a closed immersion between special fibers, it must be an isomorphism there and hence as a finite flat map its degree is 1. Thus, $f$ is an isomorphism, so $\ker f = 1$ as desired.

QED

If we work with algebraic varieties instead of schemes, then there is a positive answer: I described an ind-variety parametrising families of connected algebraic subgroups of an algebraic group $G$, cf. “Universal family of the subgroups of an algebraic group,” (arXiv) which I would like to outline. Please note that this work is a pre-print, and while some persons have read it with enthusiasm it never was formally peer-reviewd and a lot of inaccuracies and clumsy statements are probably still to be found there.

A natural idea is to replace each connected group by its Lie algebra and use the universal property of the Grassmann variety to construct our universal family. The main difficulties in this approach lie in the complex structure of the set $\mathbf A$ of $k$-dimensional algebraic subalgebras of the Lie algebra $\mathfrak g$ of $G$, that is, the set of subalgebras of $\mathfrak g$ which are the Lie algebra of an algebraic subgroup of $G$. The complexity of this set is entirely imputable to tori. For a torus, the kernel of the exponential mapping is a lattice and the closed subgroups of that torus correspond to the linear subspaces of its Lie algebra that are generated by points in this lattice. We can view them as the rational points in the complex Grassmann variety. This is also why we need ind-varieties in the construction.

Differentiating families of sub groups

The well-known study of the analytic structure of complex algebraic varieties by Serre (GAGA) allow us to see that the family the family of Lie algebras $\mathop{\mathcal{L}}(\mathcal{H})$ associated to a flat family $\mathcal{H}$ of subgroups of $G$ is also flat:

3.4 Corollary. – Let $\mathcal{H}$ be a family of connected $k$-dimensional algebraic subgroups of $G$ parametrised by $Q$. If $\mathcal{H}$ is flat, then there is a morphism $\psi:Q\to\Gamma$ such that $\mathop{\mathcal{L}}(\mathcal{H})$ is obtained by pulling back the tautological bundle $\mathcal{T}\to\Gamma$ through $\psi$, where $\Gamma$ is the Grassmann variety of $k$-dimensional subspaces of the Lie algebra of $G$.

In this setting, the image of $Q$ through $\psi$ is a constructible subset of $\Gamma$, contained in the set $\mathbf A$ of algebraic subalgebras of $\mathfrak g$. It is remarkable that each subset can be obtained this way, that is, that we can solve the problem of

Integrating families of subalgebras

Theorem. – If $P$ is a constructible subset of $\Gamma$, contained in the set $\mathbf A$ of algebraic subalgebras of $\mathfrak g$ then the closure $\bar P$ of $P$ in the Grasmmann variety $\Gamma$ is also a subset of $\mathbf A$ and the subset $$ \mathcal H(\bar P) = \left\{ (g,p) \in G \times\bar P \mid g \text{ belongs to the irreducible subgroup of $G$ with Lie algebra $p$} \right\} $$ is a flat family of algebraic subgroups of $G$ parametrised by $\bar P$ and is equal to $\overline{\mathcal H(P)}$.

We can then construct a universal family by gluing together all the $\mathcal H(\bar P)$ when $P$ runs through the maximal irreducible subvarieties of the Grassmann variety $\Gamma$ which are contained in the set $\mathbf A$ of algebraic subalgebras of $\mathfrak g$.

The main observations is that there is actually a closed condition recognising subgroups $H$ of an algebraic group: with $\theta(x,y) = xy^{-1}$ this condition can be written $\theta(H\times H) \subset H$. (Incidentally, I am quite confident I learnt this characterisation of subgroups of a group in @laurent-moret-bailly's lectures in my third year at the university.) This proves that $\overline{\mathcal H(P)}$ is a set-theoretic family of subgroups of $G$.

Thanks to Chevalley's structure theorem for algebraic groups and the observation that families of subgroups of an abelian variety are rigid, i.e. essentially reduced to a trivial family (see 6.3 and following), the difficult part in the Theorem above is to prove in the affine case that $\bar P$ is contained in the set $\mathbf A$ of algebraic subalgebras of $\mathfrak g$.

It is enough to suppose $P$ irreducible and we can approach points $p_\infty$ in the closure of $P$ by a curve $C \subset \bar P$ meeting $P$ on a dense subset of $P$. We now make the crucial remark that the Levi-Malcev decomposition is semi-continuous and that families of semi-simple subalgebras of $\mathfrak g$ are rigid, which implies that we can use $G$ operation to assume all points in $C$ to contain the semi-simple part of $p_\infty$. We are essentially left with the case where each algebra in $P$ is solvable and the exponential restricts here to a an algebraic morphism!