Groups that satisfy ${ [x,y]^2 \approx 1 }$

This variety of groups has indeed been considered in the literature. It is known that the following conditions hold for every group $G$ satisfying the identity $[x,y]^2=1$:

1. $[[x,y_1,\ldots,y_m],[x,z_1,\ldots,z_n]]=1$ for all $x,y_1,\ldots,y_m,z_1,\ldots,z_n\in G$ (see [1]).
2. $[[x_1,x_2],[x_3,x_4]]=[[x_{\pi(1)},x_{\pi(2)}],[x_{\pi(3)},x_{\pi(4)}]]$ for all $x_1,\ldots,x_4\in G$ and $\pi\in S_4$ (see [2]).
3. $[[x,y],[z,w]]=[x,y,z,w][x,y,w,z]$ (see [1]).
4. $[\gamma_2(G),\gamma_3(G)]=1$ (see [2]).
5. $[G'',G]=1$ (see [2]).
6. $G'^4=1$ (see [2]).
7. The group $G$ need not be metabelian (see [3]).
8. If $G$ is finite, then $G=P\rtimes H$, where $P$ is a normal Sylow $2$-subgroup of $G$, $H$ is abelian of odd order, and $[P,H]$ is an elementary abelian $2$-group (see [1]).

References:

1. M. Farrokhi D. G., On groups satisfying a symmetric Engel word, Ric. Mat. 65 (2016), 15–20.

2. I. D. Macdonald, On certain varieties of groups, Math. Z. 76, (1961) 270–282.

3. B. H. Neumann, On a conjecture of Hanna Neumann, Proc. Glasgow Math. Assoc. 3 (1956), 13–17.

Not every group satisfying the law $[x,y]^2=1$ is metabelian. But by Theorem 4 of McDonald, I. D. "On certain varieties of groups" Math. Z. 76 1961 270–282. every group satisfying this law has second derived subgroup in the center so it is center-by-metabelian, and the derived subgroup of exponent 4.

Note. I had a wrong answer before and did not notice Farrokhi's answer when I made corrections - one hour after his answer. Farrokhi's answer is more complete.

Here are some additional details for the answer of M. Farrokhi. In the paper " On a conjecture of Hanna Neumann", B.H.Neumann constructs a certain group $G$. There are generators $a_1,\dotsc,a_4$, and additional elements defined in terms of these as follows: \begin{align*} b_{12} &= [a_1,a_2] = [a_2,a_1] \\ b_{13} &= [a_1,a_3] = [a_3,a_1] \\ b_{14} &= [a_1,a_4] = [a_4,a_1] \\ b_{23} &= [a_2,a_3] = [a_3,a_2] \\ b_{24} &= [a_2,a_4] = [a_4,a_2] \\ b_{34} &= [a_3,a_4] = [a_4,a_3] \\ c_1 &= [a_2,b_{34}] = [a_4,b_{23}] \\ c_2 &= [a_3,b_{14}] = [a_4,b_{13}] \\ c_3 &= [a_4,b_{12}] = [a_1,b_{24}] \\ d &= [a_1,[a_2,[a_3,a_4]] = [a_2,[a_3,[a_4,a_1]] = [a_3,[a_4,[a_1,a_2]] \end{align*}

There are some relations implicit in the above equations. There are also additional relations as follows:

• $a_i^2=b_{jk}^2=c_l^2=d^2=1$
• All commutators $[a_i,b_{jk}]$ that have not already been listed, are trivial.
• $[a_i,c_j]=1$ whenever $i\neq j$, and $[a_i,d]=1$

We can define a map $\phi\colon\{0,1\}^{14}\to G$ by $$ \phi(u) = a_1^{u_1}\dotsb a_4^{u_4} b_{12}^{u_5} \dotsb b_{34}^{u_{10}} c_1^{u_{11}}c_2^{u_{12}}c_3^{u_{13}}d^{u_{14}} $$ One can check that this is bijective, and one can write formulae for the permutations of $\{0,1\}^{14}$ corresponding to right multiplication by the elements $a_i$, $b_{jk}$, $c_l$ and $d$. In particular, this proves that $|G|=2^{14}$. One can also check that $$ [b_{12},b_{34}] = [b_{13},b_{24}] = [b_{14},b_{23}] = d \neq 1, $$ so $G$ is not metabelian.

In the paper "On certain varieties of groups", Macdonald states that it is easy to verify that the above group has $[x,y]^2=1$ for all $x,y\in G$. I don't see how to prove this myself. However, I have checked it by computer for 10000 randomly chosen pairs $(x,y)$, so it must be true.