Intuition for why a group can fail to have an automorphism sending a particular element to its inverse.

Below is an arguably geometric example, it may not be what you're looking for.

For a vector space $V$ over a field $F$, let $A(V)$ be the group of affine maps on $V$ which are compositions of translations and scaling maps, i.e. the affine maps of the form $x \mapsto ax + v$, for $a \in F \setminus \{0\}$ and $v \in V$. Let $T(V)$ be the subgroup of $A(V)$ consisting of translations, i.e. the maps $x \mapsto x + v$. Note that $T(V) \cong V$ as groups. We include the translations because they encode additional structure of the action of scaling maps on $V$; if we only considered the group of scaling maps, this would just be $F \setminus \{0\}$ under multiplication, which is abelian, and thus has automorphism $g \mapsto g^{-1}$.

We will now prove that when $F = \mathbb{Q}$ or $\mathbb{F}_p$, every automorphism $\phi$ of $A(V)$ preserves scaling factor, meaning if $f \in A(V)$ is of the form $x \mapsto ax + v$, then $\phi(f)$ is of the form $x \mapsto ax + v'$. In particular, this will mean that every element of $A(V)$ with scaling factor $a \neq \pm 1$ is not sent to its inverse by any automorphism of $A(V)$.

Lemma 1: Let $m, n$ be integers which are not zero in $F$. Then for $f \in A(V)$, $f$ has the property that $fg^n = g^m f$ for all $g \in T(V)$ if and only if $f$ has scaling factor $m/n$, meaning $f$ is of the form $x \mapsto (m/n)x + v$.

Proof: If $f$ is of the given form, then for $g(x) = x + u$, clearly $$f(g^n(x)) = (m/n)(x + nu) + v = (m/n)x + v + mu = g^m(f(x))$$ so $fg^n = g^m f$. In the other direction, if $f$ satisfies $fg^n = g^m f$ for all $g \in T(V)$, then writing $f(x) = ax + v$, and taking $g(x) = x + u$ for some $u \neq 0$, we have $a(x + nu) + v = ax + v + mu$, giving $anu = mu$, hence $a = m/n$ as desired.

Lemma 2: If $F = \mathbb{Q}$ or $\mathbb{F}_p$, then every automorphism $\phi$ of $A(V)$ preserves $T(V)$, meaning $\phi(T(V)) = T(V)$.

Proof: In the case $F = \mathbb{Q}$, we can identify the subgroup $T(V)$ of $A(V)$ as the set of "divisible" elements, namely those elements $g \in A(V)$ for which, for any positive integer $n$, there is an element $h \in A(V)$ with $h^n = g$. Elements $g \in A(V)$ of the form $x \mapsto ax + v$ for $a \neq 1$ do not have this property, since there can be an $h$ with $h^n = g$ only if $a$ is an $n$-th power in $\mathbb{Q}$, and the only nonzero element of $\mathbb{Q}$ which is an $n$-th power for each $n$ is $a = 1$. It is clear that any automorphism maps divisible elements to divisible elements, and non-divisible elements to non-divisible elements, so any $\phi$ has $\phi(T(V)) = T(V)$.

In the case $F = \mathbb{F}_p$, we can identify $T(V)$ as the set of elements of order $p$ (together with the identity). For an element $g$ of the form $x \mapsto ax + v$ for $a \neq 1$, $g$ is conjugate to the map $x \mapsto ax$, and thus $g$ has order dividing $p-1$, since this latter map has order dividing $p-1$. Automorphisms preserve order, so again any $\phi$ has $\phi(T(V)) = T(V)$. [end proof]

Now, let $\phi$ be an automorphism of $A(V)$, and let $f \in A(V)$, so $f(x) = ax + v$ for some $a, v$. Since $F = \mathbb{Q}$ or $\mathbb{F}_p$, $a$ is of the form $m/n$ for some integers $m, n$ which are not zero in $F$, and thus by Lemma 1, $fg^n = g^m f$ for any $g \in T(V)$. By Lemma 2, $\phi$ preserves $T(V)$, so $\phi(f)$ also has the property that $\phi(f) g^n = g^m \phi(f)$ for all $g \in T(V)$. Then, again by Lemma 1, this means that $\phi(f)$ has scaling factor $m/n = a$, so $\phi$ preserves the scaling factor of $f$.