Existence of group $G$ with $|G|>2$ and automorphism group of odd order

Yes, there exists indeed one group explicitly known with odd order automorphism group, of order $5^7$ and exponent $125$. For a discussion with more references see this MO-question.

References: On Minimal Orders of Groups with Odd Order Automorphism Groups.

There are many examples of finite $p$-groups, $p$ a prime, whose automorphism group is a $p$-group itself. So for $p > 2$ they provide examples of the kind you are looking for.

An accessible construction is given in my paper

A. Caranti. A simple construction for a class of $p$-groups with all of their automorphisms central. Rend. Semin. Mat. Univ. Padova 135 (2016), 251-258.

The paper can be found in the arXiv.

Actually, it is currently known that the least nontrivial odd order automorphism group has order $3^7$. It was proved by Peter Hegarty and Desmond Machale in Minimal odd order automorphism groups

Another example is definitely not minimal, but still interesting enough to be mentioned. It is a complete group of order $3\cdot 7^{12}\cdot 19$, that was constructed by R.S. Dark in "A complete group of odd order" (here is the post, from which I found out about its existence: Does every finite non-trivial complete group have even order?). It also satisfies your condition as any complete group is isomorphic to its automorphism group and moreover it serves as a counterexample to a conjecture weaker than yours - the so-called Rose conjecture, that stated, that "all nontrivial finite complete groups have even order".