Question about $\operatorname{Aut}(D_\infty)\cong D_\infty$

Let's use the more intuitive presentation $$D = D_{\infty} = \langle r, f | f^2 = 1, \ frf = r^{-1} \rangle$$ which is equivalent to yours by $f \mapsto a$ and $r \mapsto ab$.

First note that $C = \langle r \rangle$ is an infinite cyclic subgroup of $D$ of index 2, hence normal. Better, it's characteristic: it comprises precisely all the elements of $D$ of infinite order, and since automorphisms preserve order, they cannot send elements of $C$ outside of $C$. Thus every automorphism of $D$ restricts to an automorphism of $C$, so if $\rho \in \operatorname{Aut}(D)$ then $\rho(r) = r^{\pm 1}$.

Next, $\rho(f) = fr^k$ for some integer $k$ without restriction, because $\rho$ must fix the coset $fC$ as a set. Thus the automorphisms are parametrized by pairs of signs and integers. You should check that such pairs indeed give automorphisms of $D$ and that they satisy the obvious composition law that makes their aggregate a dihedral group.