Existence of a normal subgroup with $\lvert\operatorname{Aut}{(H)}\rvert>\vert\operatorname{Aut}{(G)}\rvert$

A version of Robin Chapman's answer that might be easier to verify:

The additive group $H$ of a field of order $2^{n}$ has two nice types of automorphism: multiplication by a scalar, and the Frobenius automorphism. The semi-direct product $G$ is called $AΓL(1,2^{n})$. It has order $(2^{n})⋅(2^{n}−1)⋅(n) ≤ 2^{3n}$, and for $n ≥ 2$ it is its own automorphism group.

It has the additive group $H$ of the field as a normal subgroup. However, as just an additive group, $H$ has automorphism group all n×n matrices over the field with 2 elements, which has size $(2^{n}−1)⋅(2^{n}−2)⋅⋅⋅(2^{n}−2^{n−1}) ≥ 2^{(n−1)^{2}}$.

For large $n$, $|Aut(G)| = (2^{n})⋅(2^{n}−1)⋅(n) ≤ 2^{3n} ≤ 2^{(n−1)^{2}} ≤ (2^{n}−1)⋅(2^{n}−2)⋅⋅⋅(2^{n}−2^{n−1}) = |Aut(H)|$

In particular:

For $n = 4$, $G = AΓL(1,16)$, $Aut(G) = G$ has order $960$, and $Aut(H) = GL(4,2)$ has order $20160$.

For $n = 10$, $G = AΓL(1,1024)$, $Aut(G) = G$ has order $10475520$, and $Aut(H) = GL(10,2)$ has order $366440137299948128422802227200$.

In other words, $Aut(H)$ can be enormously bigger than $Aut(G)$.

This is reasonably important in finite group theory:

$G$ is a very rigid group with lots of structure. Because it contains all the "automorphisms" of the field, the group itself determines the field. An automorphism of the group will have to be an automorphism of the field, and we've already listed them all. Sometimes this expressed by saying the group G determines the geometry of the affine line on which it acts.

$H$ is a very floppy group to which you can do nearly anything. Without the maps encoding scalar multiplication, $H$ no longer remembers the field that defined it. It is just a vector space, and so instead of the (very few) field automorphisms, you are now allowed to use any vector space automorphism. $H$ has lost its structure.

I think $H$ is the canonical example of a horribly structureless group. Groups like $GL$ and $AΓL$ are pretty standard examples of groups with very clear structure. The symmetric group is very similar. Except for a few early cases, a symmetric group is its own automorphism group because it already contains within itself the set of points on which it is acting.

The quasidihedral group of order 16, QD16, has automorphism group of order 16, but it has Q8 (the quaternion group) as a normal subgroup, and | Aut(Q8) | = 24.

Steve

An elementary Abelian group can have a large automorphism group. Let $H$ be a largish elementary Abelian group, and $G$ be an extension of $H$, say be a semidirect product with a cyclic group acting faithfully on $H$. Then it is possible for Aut$(G)$ to be smaller than Aut$(H)$.