exponential/logarithm for unipotent algebraic groups

This is false in characteristic $p$, no matter how large $p$ is. The counterexample is the group parameterized by

$\begin{pmatrix} 1 & t & t^p \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}$

Its Lie algebra is generated by the matrix

$\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}$

whose exponential does not lie in the group.

EDIT: I roll back to the previous proof, in characteristic 0 only. My last proof including characteristic $p$ was false (thanks to Will Sawin for noticing this).

Let $G={\rm GL}_{n,k}$\,, where $k$ is a field of characteristic 0. Consider the formal power series $$\exp(x)=1 + x+\frac{1}{2!}x^2+\dots$$ over ${\mathbb{Q}}$ and the polynomial $$\exp_{<n}(x)=1+x+\dots+\frac{1}{(n-1)!}x^{n-1},$$ which is defined also in characteristic $p\ge n$.

Theorem 1. Let $k$ be a field of characteristic 0. Let $H\subset G={\rm GL}_{n,k}$ be an algebraic subgroup defined over $k$. Let $X\in{\rm Lie}(H)\subset{\mathfrak{gl}}_{n,k} = M_n(k)$ be a nilpotent matrix. Then $\exp_{<n}(X)\in H(k)$.

Note that since $X\in M_n(k)$ is nilpotent, we have $X^n=0$.

The version of Theorem 1 in positive characteristic is false, see the answer of Will Sawin.

Set $V=k^n$. Write $$T^{ij}(V)=V\otimes\dots\otimes V\otimes V^*\otimes\dots\otimes V^*$$ ($i$ times $V$, $j$ times $V^*$), where $V^*$ is the dual space to $V$, and set $$W=W^{\le N}=\bigoplus_{0\le i,j\le N} T^{ij}(V).$$ Let $\theta=\theta^{\le N}$ denote the natural representation of $G={\rm GL}(V)$ in $W$, and let $d\theta$ denote the corresponding representation of ${\rm Lie}(G)={\mathfrak{gl}}(V)$ in $W$. Since $X$ is nilpotent and $\theta\colon {\rm GL}(V)\to {\rm GL}(W)$ is a homomorphism of linear algebraic groups, the linear operator $(d\theta)(X)\in{\rm End}(W)$ is nilpotent; see Springer, Linear Algebraic Groups (2nd ed.), Theorem 4.4.20. Note that in general it is not true that $((d\theta)(X))^n=0$, but one can show that $((d\theta)(X))^{2nN}=0$.

Theorem 2. Assume that ${\rm char}(k)=0$. Then for any nilpotent matrix $X\in {\rm End}(V)={\mathfrak{gl}}(V)$ we have $$\exp((d\theta)(X))=\theta(\exp(X)).$$

Note that $\theta(\exp(X))$ and $\exp((d\theta)(X))$ are defined because both $X\in {\rm End}(V)$ and $(d\theta)(X)\in{\rm End}(W)$ are nilpotent operators and because ${\rm char}(k)=0$.

We deduce Theorem 1 from Theorem 2. There exists a natural number $N=N_H$ and a tensor $t=t_H\in W=W^{\le N}$ such that $H$ is the stabilizer in $G$ of the line $k\cdot t\subset W$ with respect to $\theta$ and such that ${\rm Lie}(H)$ is the stabilizer in ${\rm Lie} (G)$ of this line with respect to $d\theta$; see Springer's book, Lemmas 5.5.1 and 5.5.2. Since $X\in {\rm Lie}(H)$, we have $(d\theta)(X)\cdot t=\lambda t$ for some $\lambda\in k$, and we have $\lambda=0$ since $(d\theta)(X)$ is nilpotent. Now it follows from Theorem 2 that $$\theta(\exp(X))\cdot t=\exp((d\theta)(X))\cdot t=t,$$ and therefore, $\exp(X)\in H(k)$, which proves Theorem 1.

Proof of Theorem 2. Let $A$ be a Lie group over $k=\mathbb{R}$ or $k={\mathbb{C}}$. As usual, for $X\in{\rm Lie}(A)$ we define the exponential map $Z(s)=\exp(sX)\in A$ as the solution of the differential equation $\frac{d}{ds} Z(s)=X\cdot Z(s)$ with initial condition $Z(0)=1_A$. Then for $A={\rm GL}(V)$ the exponential map is defined by the convergent series above. If $\phi\colon A\to B$ is a homomorphism of Lie groups, then the following diagram commutes: $$ \require{AMScd} \begin{CD} {\rm Lie}(A) @>{d\phi}>> {\rm Lie}(B);\\ @V{\exp_A}VV @VV{\exp_B}V \\ A @>{\phi}>> B; \end{CD} $$ Indeed, both composite maps are solutions of the same differential equation with the same initial condition.

Note that the algebraic groups ${\rm GL}(V)$ and ${\rm GL}(W^{\le N})$ and the homomorphism $\theta=\theta^{\le N}$ are all defined over ${\mathbb{Q}}$. We use the idea of the Lefschetz principle. We consider the finitely generated field $l={\mathbb{Q}}(x_{ij})$, where $x_{ij}\in k$ for $1\le i,j\le n$ are the matrix elements of $X$. We embed $l$ into ${\mathbb{C}}$. We obtain that $$\exp((d\theta)(X))=\theta(\exp(X))$$ over ${\mathbb{C}}$. Since $X$ and $(d\theta)(X)$ are nilpotent, the expressions in the formula above are actually polynomials of $(d\theta)(X)$ and $X$, respectively. This completes the proof of Theorem 2.