There does not exist group $G$ such that Aut$(G)\cong \mathbb{Z}_n$ (for odd $n$)

It is not true that $|Aut(G)|=m!$ for some $m$. For example, if $G=\mathbb{Z}_5$, then $Aut(G)\cong\mathbb{Z}_4$ and 4 is not a factorial.

If you know that $G$ is abelian, then it is the product of cyclic group $G=\prod \mathbb{Z}_{p_i^{k_i}}$ where the $p_i$ are prime. It follows that $Aut(\mathbb{Z}_{p_i^{k_i}})$ is a subgroup of order $(p_i-1)p_i^{k_i-1}$ of $Aut(G)$. If $p_i$ is odd, then $p_i-1$ is even so the order of $Aut(G)$ is even. Similarly, if $p_i=2$ and $k\geq 2 $ then $p^{k-1}$ is even. So you just need to check the case where $G= \mathbb{Z}_2^m$ in which unless $m=1$ (and then $Aut(G)$ is trivial), you can always find an automorphism of order 2 - for example, switch two basis elements.

Actually, it is true that $Aut(G) $ is cyclic (and not trivial) iff $G$ is cyclic of order 4, $p^k$ or $2p^k$ where $p$ is prime (see here). Using that it is easy to see that the order of $Aut(G)$ must be 2 or $(p-1)p^{k-1}$ both of which are even.

$\DeclareMathOperator{\Aut}{Aut}$ We'll show the slightly stronger result that $|\Aut(G)| \not= 2n+1$ for any $n \ge 1$ EDIT: Provided $G$ is abelian: see the comments.

If $\Aut(G)$ is trivial, then there is nothing to show. Suppose, then, that $G$ has a nontrivial automorphism group. We will show that $|\Aut(G)|$ is even by explicitly exhibiting an automorphism $\phi$ of $G$ having order $2$ for any group $G$. It will then follow by Lagrange's theorem that $\Aut(G)$ has even order.

If $G$ has an element of order at least $3$, then $\phi:x \to x^{-1}$ is the desired automorphism of even order. It only remains to consider the case where $G$ contains only elements of order less than or equal to $2$. In that case, we must have that $G$ is a vector space over $\mathbb{Z}_2$ (see Group where every element is order 2 for a proof of this result). If $G$ is one dimensional as a vector space, then $\Aut(G) = \Aut(\mathbb{Z}_2)$ is trivial, so there is nothing more to show. For $G$ having dimension greater than $1$, we have that $\Aut(G) \cong \operatorname{GL}(G)$, so if we pick an ordered basis for $G$, then the automorphism which interchanges the first and second basis vectors (and fixes all others) is an order $2$ automorphism. (This argument, which Slade gave in the comments, applies even to infinite $G$, while my previous argument tacitly assumed $G$ was finite.)