Group structure of rotating-square puzzle

GAP shows that the group is in fact isomorphic to $S_5$. A geometric interpretation was requested for what $5$ things are being permuted. Consider the following $5$ sets of edges between the vertices.

$(1,2,5,4)\leftrightarrow(orange,blue,purple,green)$

$(2,3,6,5)\leftrightarrow(red,blue,purple,green)$

To see that this is all of $S_5$ and not some subgroup, compute some products of elements.

$(orange,blue,purple,green)*(red,blue,purple,green)=(red,purple,orange,blue,green)$

which has order $5$, so the order of the group is a multiple of $5$.

$(orange,blue,purple,green)*(red,blue,purple,green)^{-1}=(red,orange,blue)$

which has order $3$, so the order of the group is a multiple of $3$.

And $(orange,blue,purple,green)$ has order $4$ so the order of the group is a multiple of $4$.

Now the order of the group must be a multiple of $3*4*5=60$, so either $S_5$ or $A_5$. But we have elements of order $4$ in our group, which leaves only $S_5$.

Here are a few tips for identifying the isomorphism type of a small group like this one:

• Determine generators.
• Determine the size of the group.
• Determine the orders of its elements.
• Determine relations between the generators.

Once you have done this, you can consult a list of small groups to get a short list of candidates. You already know that $G$ is isomorphic to a subgroup of $S_6$, so you can consult a list of subgroups of $S_6$ right away, for example here.

Here's a start from my end; you have observed $G=\langle(1\ 2\ 5\ 4),(2\ 3\ 6\ 5)\rangle$. Because these two $4$-cycles do not commute, this implies $|G|$ is a multiple of $8$. Moreover, the product $$(1\ 2\ 5\ 4)(2\ 3\ 6\ 5)=(1\ 2\ 3\ 6\ 4),$$ shows that $G$ has an element of order $5$, so its order is a multiple of $40$. Trying out another combination of the generators yields $$(1\ 2\ 5\ 4)(2\ 3\ 6\ 5)^2=(1\ 2\ 6\ 5\ 3\ 4),$$ so $|G|$ is a multiple of $6$ as well. The linked document then leaves only three isomorphism types for $G$; either $S_5$, $A_6$ or $S_6$. Because both generators are odd permutations, it cannot be $A_6$. Then it is either $S_5$ or $S_6$, and you could ask whether every permutation of the vertices can be achieved.

More simply, since $G$ contains the $6$-cycle $$(1\ 2\ 6\ 5\ 3\ 4)=(3\ 6\ 4\ 5)(1\ 2\ 3\ 4\ 5\ 6)(3\ 5\ 4\ 6),$$ and since $(1\ 2\ 3\ 4\ 5\ 6)$ and any transposition together generate $S_6$, you could ask whether $G$ contains any element of the form $(3\ 6\ 4\ 5)\tau(3\ 5\ 4\ 6)$, where $\tau$ is a transposition.

---

This is clearly a very ad hoc approach, and some familiarity with small groups goes a long way in making the right considerations. Using a software package such as GAP can be a very helpful alternative.