Examples of subgroups where it's nontrivial to show closure under multiplication?
A particularly nice example is the following : suppose the finite group $G$ acts on the finite set $X$ in such a way that every nontrivial element of $G$ has at most one fixed point. Let $S$ be the set of elements of $G$ that have no fixed points. Then $H=S\cup \{1\}$ is a subgroup of $G$.
I believe the only known proofs are representation-theoretic (or at least that was the case at first).
Consider the symmetric group $S_n$ on $n\geq 2$ letters. The alternating group $A_n$ is the subgroup of $S_n$ given by all even permutations of $S_n$.
One proof uses the signum or sign function $s:S_n\rightarrow\{\pm 1\}$ which assigns to a permutation $\pi$, $+1$ if $\pi$ is even, and $-1$ if $\pi$ is odd. It can be shown that the sign function is a homomorphism, i.e., $s(\pi\sigma) = s(\pi)\cdot s(\sigma)$.
It follows that the product of two even permutations is even and so the multiplication in $A_n$ is well-defined.
Let $G=GL(4,k)$ (the group of all $4\times4$ invertible matrices with entries in a field $k$) and let $N$ be the subgroup of those matrices $M\in GL(4,k)$ of the form$$\begin{bmatrix}a_{11}&a_{12}&a_{13}&a_{14}\\a_{21}&a_{22}&a_{23}&a_{24}\\0&0&a_{33}&a_{34}\\0&0&a_{43}&a_{44}\end{bmatrix}.$$