Groups with given automorphism groups
Ledermann and B.H.Neumann ("On the Order of the Automorphism Group of a Finite Group. I", Proc. Royal Soc. A, 1956) have shown the following:
Theorem. Let $n > 0$. There exists a bound $f(n)$ such that if $G$ is a finite group with $|G| \geq f(n)$, then $|\operatorname{Aut}(G)| \geq n$.
An immediate consequence is that up to isomorphism, there are only finitely many finite groups $G$ with $|\operatorname{Aut}(G)| \leq n$. Hence for any finite group $X$, up to isomorphism there are only finitely many finite groups $G$ with $\operatorname{Aut}(G) \cong X$.
Among infinite groups this is no longer true, and indeed there are infinitely many groups $G$ with $\operatorname{Aut}(G) \cong \mathbb{Z} / 2 \mathbb{Z}$.
Then there is of course the question of determining all finite groups $G$ with given automorphism group $\operatorname{Aut}(G) \cong X$. For this, see for example
Iyer, Hariharan K. On solving the equation Aut(X)=G. Rocky Mountain J. Math. 9 (1979), no. 4, 653–670.
This paper gives a solution to the problem in some cases, and determines for example all $G$ with $\operatorname{Aut}(G) \cong S_n$. There is also a different proof of the fact that there are only finitely many groups with a given automorphism group (Theorem 3.1 there).
Mikko's nice answer concerns finite groups $G$. Let me here answer for infinite groups $G$ (but still finite automorphism groups, as in the question).
The picture is indeed very different:
For $A=C_2$ cyclic, there exists uncountably many non-isomorphic (abelian countable) groups $G$ with $\mathrm{Aut}(G)\simeq C_2$.
Indeed, for $I$ a set of primes, let $B_I$ be the additive subgroup of $\mathbf{Q}$ generated by $\{1/p:p\in I\}$. Then $B_I$ and $B_J$ are isomorphic if and only if the symmetric difference $I\triangle J$ is finite, and $\mathrm{Aut}(B_I)=\{1,-1\}$ (easy exercise: more generally for a nonzero subgroup $B$ of $\mathbf{Q}$, its automorphism group is $\{t\in\mathbf{Q}^*:tB=B\}$ acting by multiplication).
One also gets the group $C_2^n$ ($n\ge 1$) in a similar fashion. Say, for $n=2$, choose $I,J$ such that both $I\smallsetminus J$ and $J\smallsetminus I$ are infinite: then $\mathrm{Aut}(B_I\times B_J)\simeq C_2\times C_2$.
In general, if a group $G$ has finite automorphism group $A$, then its center has finite index in $G$, because $G/Z(G)$ embeds into $A$. A well-known result then implies that $[G,G]$ is finite.
[Also, it follows that if $A$ is cyclic of odd order, we deduce that $G/Z(G)$ is cyclic, and hence $G$ is abelian, and then $G$ has to be a finite elementary abelian $2$-group, and then $G=1$ or $G\simeq C_2$, whence $A=1$. In other words, for no group (finite or infinite) $G$, $\mathrm{Aut}(G)$ is cyclic of odd order $>1$.]
One more example to mention that one gets non-abelian groups: let $F$ be a finite group. Then for every torsion-free abelian group $B$, $\mathrm{Aut}(B\times F)$ is a semidirect product $(\mathrm{Aut}(F)\times\mathrm{Aut}(B))\ltimes\mathrm{Hom}(B,Z(F))$. If $\mathrm{Aut}(B)=\{\pm 1\}$, then the $\mathrm{Aut}(B)$-action on $\mathrm{Hom}(B,Z(F))$ is trivial and this reduces to the product $\mathrm{Aut}(B\times F)=(\mathrm{Aut}(F))\ltimes\mathrm{Hom}(B,Z(F))\times\mathrm{Aut}(B)$. For $B=B_I$, we have $\mathrm{Hom}(B_I,Z(F))\simeq Z(F)$. For instance, for $F=C_2$ one gets $\mathrm{Aut}(B_I\times C_2)\simeq C_2^2$. The smallest non-abelian group we can get this way has order 12, namely for $F=C_3$ or $F=D_6$ (dihedral group of order 6), one gets $\mathrm{Aut}(B_I\times F)\simeq D_6\times C_2$. For $F=C_4$ one gets $\mathrm{Aut}(B_I\times C_4)\simeq D_8\times C_2$.
I don't know if we can obtain abelian $\mathrm{Aut}(B_I\times F)$ when $|F|\ge 3$. This holds if and only if $\mathrm{Aut}(F)$ is abelian and acts trivially on $F$. Then $F$ is non-abelian, of nilpotency class 2. Possibly some large $p$-groups satisfy this (see Jain-Rai-Yadav (arXiv link) for a discussion of large $p$-groups with abelian automorphism groups; however they don't indicate if they can be chosen so that automorphisms are trivial on the center).