Examples of finite nonabelian groups.
Here's a bunch of examples I've collected from my notes, group theory texts, and various places around the Internet. This has become somewhat of a treatise, but nonetheless, I hope you and others enjoy them.
Familiar generalizations.
Generalized dihedral groups, denoted $\mathcal{D}(A)$ or $\operatorname{Dih}(A)$, are formed by letting an involution (element of order $2$) act on an arbitrary abelian group $A$ by inversion. So, $D_{2n}=\operatorname{Dih}(\mathbb{Z}/n\mathbb{Z})$.
Another sister of the dihedral groups are the semidihedral groups. These are all $2$-groups which behave just a tiny bit different from their dihedral counterparts (see the presentation).
The quaternion group $Q_8$ (not to be confused with the related-but-not-identical quaternion algebra) is a Hamiltonian group of order $8$. It's made of three normal subgroups $\langle i \rangle$, $\langle j \rangle$, and $\langle k \rangle$, which intersect at the order $2$ center $Z(Q_8)=\langle -1 \rangle$. $Q_8$ is the first counterexample you should try whenever you think anything might be true. (one example)
The generalized quaternion groups, denoted $Q_{4n}$ for $n\geq 2$, naturally extend the group presentation of $Q_8$ to higher orders. The important thing about generalized quaternion groups is that they have a unique element of order $2$, whereas all other $p$-groups with unique elements of order $p$ are cyclic.
These can be most easily understood as the quotient of the semidirect product $\mathbb{Z}_{2^n}\rtimes \mathbb{Z}_4$ by the subgroup $\langle(2^{n-1},2)\rangle$.
You can also realize $Q_{4n}$ as the subgroup of $\mathbb{H}^\times$ generated by $\cos(\pi/n)+\mathbf{i}\sin(\pi/n)$ and $\mathbf{j}$, where $\mathbb{H}$ denotes the (division) ring of Hamiltonian quaternions (with generators $\mathbf{i},\mathbf{j},$ and $\mathbf{k}$, under the usual relations). I found this construction very interesting
I'll stop here for a moment to note that the dihedral groups, semidihedral groups, and generalized quaternion groups constitute all $2$-groups of maximal class, which makes them super important to know for people who study $p$-groups.
- We can even generalize the generalized quaternions. Replacing $\mathbb{Z}_{2^n}$ in the above definition with an arbitrary cyclic group $A$ of even order $n$ gives us the dicyclic groups, denoted $\operatorname{Dic}(A)$ or $\operatorname{Dic}_n$. We can get even further into outer space and define the generalized dicyclic groups $\operatorname{Dic}(A,y)$ by allowing $A$ to be any abelian group with even order. (For generalized dicyclic groups one must specify which element $y\in A$ of order $2$ behaves like the unique element of order $2$ in the generalized quaternion group. You can read about that here.)
Matrix groups.
$\operatorname{GL}_n(\mathbb{F}_q)$ is the group of linear transformations of an $n$-dimensional vector space over a finite field $\mathbb{F}_q$. The special linear group $\operatorname{SL}_n(\mathbb{F}_q)$ is the kernel of the homomorphism $\operatorname{det}:\operatorname{GL}_n(\mathbb{F}_q)\rightarrow \mathbb{F}_q^{\hspace{1pt}\star}$.
$\operatorname{PGL}_n(\mathbb{F}_q)$ and $\operatorname{PSL}_n(\mathbb{F}_q)$ are their projective counterparts, which essentially removes effects of scalars from $\operatorname{GL}_n(\mathbb{F}_q)$ and $\operatorname{SL}_n(\mathbb{F}_q)$ to focus on their structure. $\operatorname{PSL}_n(\mathbb{F}_q)$ is simple for $n\geq 3$ or $q\geq 4$ (which means that $\operatorname{SL}_n(\mathbb{F}_q)$ is quasisimple).
General semilinear groups, written $\Gamma\!\operatorname{L}_n(\mathbb{F}_q)$, are split extensions of $\operatorname{GL}_n(\mathbb{F}_q)$ by the Galois group of $\mathbb{F}_q$ over its prime subfield. These have projective counterparts too.
Unitriangular matrix groups (upper triangular matrices with $1$'s on the diagonal) and symplectic groups are other good matrix groups to know.
Scary groups.
- The sporadic groups are the scariest things which lurk outside of the happy little bubble that is solvable groups. This is where the Monster lives. In most cases, understanding the structure of these groups requires some esoteric machinery, so I would avoid these until you're really good at group theory.
By the way, any nonabelian simple group is an alternating group, a $\operatorname{PSL}$ or other group of Lie type, or a sporadic group. Figuring that out took about $110$ years of focused sorcery, which is fun to read about if you like math history.
Frobenius groups.
- Frobenius groups are made of two parts: a normal subgroup $N$, called the Frobenius kernel, which is acted upon by a subgroup $K$, the Frobenius complement, with the defining property that $K$ acts on $N$ fixed point freely - that is, $n^k\ne n$ for all non-identity elements $n\in N,k\in K$. One example is $S_3$, for which the kernel is generated by the $3$-cycles and the complement is generated by (your choice of) a $2$-cycle. The structure of Frobenius groups is well understood - as it turns out, $N$ must be nilpotent, and the Sylow subgroups of $K$ must be either cyclic or generalized quaternion.
Frobenius groups are also the subject of a major theorem in representation theory which proves an alternative definition is equivalent to the standard one. I personally don't find this alternative definition particularly enlightening, but the proof of this theorem demonstrates how powerful representation theory can be.
- $2$-Frobenius groups are groups $G$ which have some subgroup $F$ which is a Frobenius group with kernel $K$ which is normal in $G$ for which $G/K$ is also a Frobenius group. An example of this is $S_4$, where $F=A_4$ (the subgroup $K$ generated by the permutions $(12)(34)$ and $(13)(24)$ are acted upon fixed point freely by $\langle (123) \rangle$) and $G/K\cong S_3$.
Frobenius and $2$-Frobenius groups are important for element orders. Whenever a solvable group $G$ lacks an element of order $pq$, where $p$ and $q$ are primes dividing $|G|$, the Hall $\{p,q\}$ subgroups of $G$ (subgroups of order $p^aq^b$ with index coprime to their order) are either Frobenius or $2$-Frobenius.
$2$-Frobenius groups were also important in the proof of the Odd Order theorem, which derived a contradiction by showing that a non-solvable group with odd order would have to contain certain $2$-Frobenius groups in a way that was not possible.
- Zassenhaus groups are somewhat related to Frobenius groups in that they are defined by counting fixed points of their action on a set. (I'll include more info on these soon.)
Cool families of $p$-groups.
$p$-groups are groups of prime power order $p^n$. We already saw some families of $2$-groups in the first section; here, we will discuss families of $p$-groups of where $p$ is not necessarily even.
- A special group is a $p$-group $G$ in which the center, derived, and Frattini subgroups coincide ($Z(G)=G^\prime=\Phi(G)$). If these subgroups are cyclic of order $p$, we call the group extraspecial. The combination of these properties gives rise to some very interesting representation-theoretic behavior. In special groups, the commutator map from $G/Z(G)\times G/Z(G)\rightarrow G^\prime$ satisfies $[g,hk]=[g,h][g,k]$, so in extraspecial groups this may be considered as bilinear form over $\mathbb{F}_p$ (which, as it turns out, is skew-symmetric, alternating, and non-singular). If $p$ happens to be $2$, we can additionally show that the squaring map is a quadratic form. You can read more about this in $\oint 3.10$ of Wilson's Finite Simple Groups, which can be seen in a preview on the web. Note also that it is particularly easy to classify the conjugacy classes of extraspecial groups, so they often make good testing grounds for conjectures in representation theory.
Some favorites from my toolbox.
$M_{16}$, the group with the coolest name, is the smallest example of a group with two distinct isomorphic characteristic subgroups, and thus another great counterexample group. $M_{16}$ is pretty close to being abelian - it is just $\mathbb{Z}_2\times \mathbb{Z}_8$ with a little "twist."
The octahedral group is the symmetries of an octahedron. I'm partial to the binary octahedral group $2\mathcal{O}$, which has order $48$ and is constructed by replacing certain elements of $\operatorname{GL}_2(\mathbb{F}_3)$ with scalar multiples in $\operatorname{GL}_2(\mathbb{F}_9)$. (I recently heard from a coauthor that Marty Isaacs refers to this group "fake $\operatorname{GL}(2,3)$", which I think is hilarious.)
$\Gamma$ groups are the affine transformation version of $\Gamma\operatorname{L}_n(\mathbb{F}_q)$. I like taking small subgroups of one dimensional $\Gamma$ groups when I want a very nonabelian group in which I can still easily write out arithmetic.
My second favorite matrix group is the Valentiner group, isomorphic to $\operatorname{PGL}_3(\mathbb{F}_4)$. My favorite sporadic group is the Thompson group.
I'm a big fan of making Frobenius groups of the form $\mathbb{Z}_{p}\rtimes \mathbb{Z}_q$ (and $2$-Frobenius groups of the form $(\mathbb{Z}_{p}\rtimes \mathbb{Z}_q)\rtimes \mathbb{Z}_r$), where $p,q,r$ are primes (necessarily) satisfying $p\equiv 1 \pmod q$ (and $p\equiv 1 \pmod {qr})$. Need an easy nonabelian group of order $21$? Boom, done.
Another particularly nice, easy class of Frobenius groups are those of the form $C_p\rtimes C_{p-1}$, where $p$ is an odd prime, also known as the holomorph of $C_p$, $\operatorname{Hol}(C_p)$. These groups arise naturally in Galois theory as the Galois group of $\mathbb{Q}\left(2^{1/p},\xi\right)$, the splitting field of $x^p-2$ over $\mathbb{Q}$. (Here, $\xi$ is a $p^\text{th}$ root of unity.) In this case, the most obvious set of generators is $$\sigma:\left\{\begin{array}{l}2^{1/p}\mapsto \xi 2^{1/p}\\ \xi\mapsto \xi\end{array}\right.\hspace{15pt}\text{and}\hspace{15pt}\tau:\left\{\begin{array}{l}2^{1/p}\mapsto 2^{1/p}\\ \xi \mapsto \xi^m\end{array}\right.$$ where $m$ has order $p-1$ mod $p$.
For an abstraction of the previous example, check out Z-groups, groups with all Sylow subgroups are all cyclic. These are related to extraspecial groups and Zassenhaus groups, and tend to be good examples in character theory. Like most things with well-understood character theory, Z-groups came up a lot in the papers leading up to the Classification theorem.
Double and triple covers of alternating groups show up as counterexamples in finite group theory often.
Make your own.
You can take wreath products of any of the above nonabelian groups to get very strange looking groups indeed. (Just the other day $(S_p\wr S_p)\wr S_p$ came up in chat.)
If you know something about the automorphisms of a group $G$, you can take semidirect products $G\rtimes H$ with other groups $H$ (usually those that embed into $\operatorname{Aut}(G)$). In particular, holomorphs can be an easy way to make new groups with easy arithmetic.
If two groups $H$ and $K$ have isomorphic central subgroups $Z_1\leq Z(H)$ and $Z_2\leq Z(K)$, you can take the (external) central product of $H$ and $K$, written $H{\small\text{ Y }} K$. This is done by identifying points in $Z_1$ with the corresponding points in $Z_2$, then taking the quotient of $H\times K$ by this diagonal central subgroup. (Explicitly, if $\theta:Z_1\rightarrow Z_2$ is an isomorphism, let $Z=\{(z,\theta(z)^{-1}):z\in Z_1\}$, and $H{\small\text{ Y }} K=\left(H\times K\right)/Z$.) There's a rich theory behind central products, and you can make some nifty things happen with them. (example)
If you want to give a group theorist a headache, take a direct product with one component from each of the above categories, wreath it with itself, and ask about the structure of its Sylow $2$-subgroup.
The 168 element symmetry group of the Fano plane, which is the smallest example of a simple group that is neither abelian (cyclic) nor covered by meh's answer (alternating). And it is of course onlly the smallest member of a bigger family ;)
The symmetry group of the rubiks cube is non-abelian, otherwise it would be easy to solve one!