Does $\mathrm{Aut}(\mathrm{Aut}(...\mathrm{Aut}(G)...))$ stabilize?
I remember that my old grad classmate from Berkeley, Joel Hamkins, worked on the transfinite version of this problem. The Automorphism Tower Problem, by Simon Thomas, is an entire book on this subject. The beginning of the book gives the example of the infinite dihedral group $D_\infty$, in the sense of $\mathbb{Z}/2 \ltimes \mathbb{Z}$. It says that the automorphism tower of this group has height $\omega+1$. It also treats Joel's theorem, which says that every automorphism tower does stabilize, transfinitely. A Proceedings paper with the same author and title says that Wielandt showed that every finite centerless group has a finite automorphism tower.
An improved answer: Simon's book later shows that the automorphism tower of the finite group $D_8$ has height $\omega+1$, and that for general finite groups no one even knows a good transfinite bound. (The $8$ may look like a typo for $\infty$, but it's not :-).) Apparently the centerless condition is essential in Wielandt's condition.
Also, to clarify what these references mean by the automorphism tower, they specifically use the direct limit of the conjugation homomorphisms $G \to \mbox{Aut}(G)$. $D_8$ is abstractly isomorphic to its automorphism group. This is a different version of the question that I suppose does not have a transfinite extension. Section 5 of Thomas' book implies that it's an open problem whether the tower terminates in this weaker sense, for finite groups.
Finally an arXiv link to Joel Hamkins' charming paper, Every group has a terminating transfinite automorphism tower.
As other people in this thread have pointed out, it's unsatisfying to make an automorphism tower that only stabilizes transfinitely as a direct limit, when all of the finite terms of the tower are abstractly isomorphic to the base group $G$. I Googled around a bit more and came back to the same two sources, Thomas' book, and this time a joint result of Hamkins and Thomas which is in chapter 8 of the book.
If an automorphism tower stabilizes after exactly $n \in \mathbb{N}$ steps in the direct limit sense, then it also stabilizes after exactly $n$ steps in the weaker abstract isomorphism sense. (Otherwise the direct limit "wouldn't know to stop".) Hamkins and Thomas do better than that. For any two ordinals $\alpha$ and $\beta$, which may or may not be finite numbers, they find one group $G$ whose automorphism tower has height $\alpha$ and $\beta$ in two different models of ZFC set theory. (Whether it's really the "same" group in different worlds is unclear to me, but their models are built to argue that it is so.) I would suppose that it is possible to make a tower without isomorphic terms by taking a product of these groups, even without the two-for-one property.
Other than one paper on the Grigorchuk group by Bartholdi and Sidki, I haven't found anything on automorphism towers of finitely generated groups. The Grigorchuk group has a countably infinite tower, but I'd have to learn more to know whether the terms are abstractly isomorphic.
"if G is a non-abelian simple group, then Aut(Aut(G))=Aut(G)."
This is Theorem 7.14 (Page 162) of
Rotman, Joseph J. (1995), An introduction to the theory of groups,
Berlin, New York: Springer-Verlag, ISBN 978-0-387-94285-8
© 1995 Springer-Verlag New York, Inc.
ISBN 0-387-94285-8 Springer-Verlag New York Berlin Heidelberg
ISBN 3-540-94285-8 Springer-Verlag Berlin Heidelberg New York
All the best,
Fernando.
$\DeclareMathOperator{\Aut}{Aut}$I don't know about non-stabilizing, but rigidity provides many examples that stabilize quickly.
1) Let $\pi$; be the fundamental group of a finite volume hyperbolic manifold $M$ of dimension $\ge 3$ with no symmetries (that is, no nontrivial self-isometries). Negative curvature implies that $\pi$; is centerless, so the map $\pi\to \Aut(\pi)$ is injective. Mostow-Prasad rigidity says that $\operatorname{Out}(\pi) = \operatorname{Isom}(M)$, so the lack of isometries implies that $\operatorname{Out}(\pi)$ is trivial and $\Aut(\pi) = \pi$. [This works verbatim for lattices in higher-rank semi-simple Lie groups subject to appropriate conditions.]
2) Let $\pi=F_d$ be a free group of rank $2\le d\lt \infty$. Then $\Aut(F_n)$ is a much larger group; however, Dyer-Formanek showed that $\operatorname{Out}(\Aut(F_n))$ is trivial. Thus since $\Aut(F_n)$ is clearly centerless, we have $\Aut(\Aut(F_n)) = \Aut(F_n)$.
3) Interpolating between these two examples, if $\pi=\pi_1(S_g)$ is the fundamental group of a surface of genus $g\ge 2$, then $\Aut(\pi)$ is the so-called "punctured mapping class group" $\text{Mod}_{g,^*}$, which is much bigger than $\pi$. Ivanov proved that $\operatorname{Out}(\text{Mod}_{g,^*})$ is trivial, and since $\text{Mod}_{g,^*}$ is again centerless, we have $\Aut(\Aut(\pi_1(S_g))) = \Aut(\pi_1(S_g))$.
In each of these cases, rigidity in fact gives stronger statements: Let $H$ and $H'$ be finite index subgroups of $G = \Aut(F_n)$ or $\text{Mod}_{g,^*}$. (This class of groups can be widened enormously, these are just some examples.) Then any isomorphism from $H$ to $H'$ comes from conjugation by an element of $G$, by Farb-Handel and Ivanov respectively. In particular, $\Aut(H)$ is the normalizer of $H$ in $G$. Rigidity gives the same conclusion for $H = \pi_1(M)$ as in the first example and $G = \operatorname{Isom}(H^n)$ [which is roughly $\operatorname{SO}(n,1)$]. It seems that by carefully controlling the normalizers, you could use this to construct examples that stabilize only after $n$ steps, for arbitrary large $n$.
Edit: I find the examples of $D_8$ and $D_\infty$ unsatisfying because even though $\operatorname{Inn}(D)$ is a proper subgroup of $\Aut(D)$, we still have $\Aut(D)$ isomorphic to $D$. Here is a general recipe for building similarly liminal examples. Let $G$ be an infinite group with no $2$-torsion so that $\Aut(G) = G$ and $H^1(G;\Bbb{Z}/2\Bbb{Z}) = \Bbb{Z}/2\Bbb{Z}$. (Edited: For example, by rigidity, any hyperbolic knot complement with no isometries has these properties; by Thurston, most knot complements are hyperbolic.) The condition on the $2$-torsion implies that for any automorphism $G \times \Bbb{Z}/2\Bbb{Z} \to G \times \Bbb{Z}/2\Bbb{Z}$, the composition
$$G \to G \times \Bbb{Z}/2\Bbb{Z} \to G \times \Bbb{Z}/2\Bbb{Z} \to G$$
is an isomorphism. From this we see that $\Aut(G \times \Bbb{Z}/2\Bbb{Z}) / G = H^1(G;\Bbb{Z}/2\Bbb{Z}) = \Bbb{Z}/2\Bbb{Z}$. By examination the extension is trivial, and thus $\Aut(G \times \Bbb{Z}/2\Bbb{Z}) = G \times \Bbb{Z}/2\Bbb{Z}$. However, the image $\operatorname{Inn}(G \times \Bbb{Z}/2\Bbb{Z})$ is the proper subgroup $G$.
Comments: looking back, this feels very close to your original example of $\Bbb R \times \Bbb Z/2\Bbb Z$. Interesting that it's (seemingly) much harder to find group-theoretic conditions to force the behavior the way you want, while topologically it's easy.
Also, if you instead take $G$ with $H^1(G;\Bbb Z/2\Bbb Z)$ having larger dimension, say $H^1(G;\Bbb Z/2\Bbb Z) = (\Bbb Z/2\Bbb Z)^2$, this blows up quickly. You get $\Aut(G \times \Bbb Z/2 \Bbb Z) = G \times (\Bbb Z/2\Bbb Z)^2$, but then $\Aut(\Aut(G \times \Bbb Z/2\Bbb Z))$ is the semidirect product of $H^1(G;(\Bbb Z/2\Bbb Z)^2) = (\Bbb Z/2\Bbb Z)^4$ with $\Aut(G) \times \Aut((\Bbb Z/2\Bbb Z)^2) = G \times \operatorname{GL}(2,2)$. Already the next step seems very hard to figure out. However, if you had enough control over the finite quotients of $G$, perhaps you could show that the linear parts of these groups don't get "entangled" with the rest, so that the automorphism groups would act like a product of $G \times (\Bbb Z/2\Bbb Z)^n$ with something else, with n going to infinity. If so, this could yield an example where the isomorphism types of the groups never stabilize.