$f(x) = x^6, g(x) = x^{10}$ endomorphisms $\implies G$ is abelian

This is an extended comment rather than an answer.

The assumption that $f$ is injective is necessary.

Indeed, the article Abelian Forcing Sets by Joseph A. Gallian and Michael Reid contains the following result:

Definition: a set of integers $T$ is called abelian-forcing if for any group $G$, if the map $x\mapsto x^t$ is an endomorphism for every $t\in T$ then $G$ has to be abelian.

Theorem: a set of integers $T$ is abelian-forcing if and only if the gcd of the numbers $t(t-1)$ where $t$ runs over $T$ is equal to $2$.

In the present case, the $\gcd (5\times 6,\ 9\times 10)=30\neq 2$.


The hypotheses are satisfied in any group of exponent $5$ (because $f(x)=x$ and $g(x)=1$ for all $x \in G$), and there are nonabelian groups of exponent $5$, so the claimed result is not correct.

Perhaps you should be assuming that both $f$ and $g$ are injective.


Here is a solution assuming $f$ and $g$ are both injective, which is necessary as Derek points out.

Let us say a group is $(a,b)$-abelian for integers $a,b$ if $x^ay^b=y^bx^a$ for all $x,y\in G$. It is easily checked that if a group is $n$-abelian, it is $(n,n-1)$-abelian.

Next, we claim the following.

Lemma 1. Let $G$ be a group. If $G$ is $m$-abelian with $x\mapsto x^m$ an injective endomorphism, and $G$ is $(da,d'b)$-abelian where $d$ and $d'$ divide $m$, then $G$ is also $(a,b)$-abelian.

To see this, note that $G$ must be $(ma,mb)$-abelian, so that $$(x^ay^bx^{-a}y^{-b})^m= x^{ma}y^{mb}x^{-ma}y^{-mb}=1$$ for all $x,y\in G$ and thus by injectivity $G$ is $(a,b)$-abelian, this verifies the claim.

Now in our case, we have that $G$ is $(6,5)$-abelian, so applying the lemma for $m=6$, we have that $G$ is $(1,5)$-abelian. Applying the lemma again for $m=10$, we get that $G$ is $(1,1)$-abelian, hence the result.


Actually, there is no point in restricting to $6$ and $10$ so let us show the following.

Proposition. Let $G$ be a group. If $x\to x^m$ and $y\to y^n$ are injective endomorphisms on $G$ with $\mathrm{gcd}(m-1,n-1)\mid (mn)^k$ for some $k$ then $G$ is abelian.

To prove this, we need one more lemma.

Lemma 2. If $G$ is $(a,b)$-abelian and $(c,d)$-abelian, it is $(r,s)$-abelian where $r=\gcd(a,c)$ and $s=\mathrm{lcm}(b,d)$.

Indeed, choose integers $k,l$ such that $ka+lc=r$ and note that for all $x,y\in G$ we have $$x^{ak+cl}y^s=x^{ak}y^sx^{cl}=y^sx^{ak+cl}.$$

To prove the proposition, note that $G$ is $(m,m-1)$- and $(n,n-1)$-abelian, hence $(1,(m-1))$- and $(1,(n-1))$-abelian by lemma 1. Lemma $2$ implies $G$ is $(1,(m-1)(n-1))$ abelian and so applying lemma 1 several times we get the proposition.