To show that group G is abelian if $(ab)^3 = a^3 b^3$ and the order of $G$ is not divisible by 3

First note that given condition says that $ f: G \to G$ defined as $x \to x^3$ is an injective homomorphism of $G$.

Further Note that $$ \forall a,b \in G: \quad ababab = (ab)^{3} = a^{3} b^{3} = aaabbb. $$ Hence, $$ \forall a,b \in G: \quad baba = aabb, \quad \text{or equivalently}, \quad (ba)^{2} = a^{2} b^{2}. $$ Using this fact, we obtain \begin{align} \forall a,b \in G: \quad (ab)^{4} &= [(ab)^{2}]^{2} \\ &= [b^{2} a^{2}]^{2} \\ &= (a^{2})^{2} (b^{2})^{2} \\ &= a^{4} b^{4} \\ &= aaaabbbb. \end{align}

• On the other hand, \begin{align} \forall a,b \in G: \quad (ab)^{4} &= abababab \\ &= a (ba)^{3} b \\ &= a b^{3} a^{3} b \\ &= abbbaaab. \end{align}

• Hence, for all $ a,b \in G $, we have $ aaaabbbb = abbbaaab $, which yields $$ f(ab) = a^{3} b^{3} = b^{3} a^{3} = f(ba). $$ As $ f $ is injective, we conclude that $ ab = ba $ for all $ a,b \in G $.Hence $G$ is an abelian group

---

Added: I think it's worth mentioning that there exist nonabelian group $G$ for which $x \to x^3$ is a group homomorphism.Smallest such example is Heisenberg group of order $27$ which can be thought of as all $3 \times 3$ upper diagonal matrices with $1's$ on the diagonal and other entries in the field of order $3$.As $G$ is of exponent 3 ( i.e. $x^3=1$ for all $x \in G$) hence $f$ is a homomorphism and $G$ is clearly non abelian because for example following two matrices don't commute: $$\left(\begin{array}{ccc} 1 & 1 & 0\\ 0 & 1 & 2\\ 0 & 0 & 1 \end{array}\right),$$ $$\left(\begin{array}{ccc} 1 & 1 & 0\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{array}\right)$$

In particular this also shows that the condition that $3$ does not divide order($G$) is necessary.