What is this subgroup of $\mathfrak S_{12}$?

The invariance group of the solutions set can be given a geometric interpretation as follows. Note that $\mathfrak{S}_4 \times \frac{\mathbf{Z}}{2\mathbf{Z}}$ is none other than the group of isometries of the cube.

It is known that if one cuts a cube by the bisecting plane of a space diagonal, the cross-section is a regular hexagon (see the picture at the middle of this page). The vertices of this hexagon are midpoints of (some) edges of the cube. Let $X$ be the set of corners and middles of this hexagon (it has cardinality $12$). Let us consider the following bijection between $X$ and the set $E$ of edges of the cube : if $[AB]$ is a side of the hexagon, with midpoint $M$, we map $A$ (resp. $B$) to the unique edge $e_A$ (resp. $e_B$) in $E$ containing it, and we map $M$ to the unique edge $e_M \in E$ such that $e_A$, $e_B$ and $e_M$ meet at a common vertex of the cube.

Given any solution of the initial problem, we can label the edges of the cube using the above bijection. This labelling has the following nice property : the sum of three edges meeting at a common vertex is always 21. Proof : by construction, six of these eight summing conditions are satisfied. The remaining two conditions read $b+f+j=d+h+\ell=21$ using Denis' notations, and are implied by the first six conditions.

So we found an equivalent ($3$-dimensional) formulation of the problem, namely labelling the edges of a cube. It is now clear that the symmetry group of the cube acts on the set of solutions. It remains to prove that the solution is unique up to isometry, which can be done by hand, here is how I did it : note that only two possible sums involve $1$ (resp. $13$), namely $1+8+12$ and $1+9+11$ (resp. $2+6+13$ and $3+5+13$). Therefore $1$ and $13$ must sit on opposite edges. Then $4$ and $10$ must sit on the unique edges which are parallel to $1$ and $13$. It is the easy to complete the cube.

The resulting labelling has some amusing properties For example, the sum of edges of a given face is always $28$. The sum of two opposite edges is always $14$. Finally, the sum of edges along a cyclohexane-like circuit is always $42$.

It is quite easy to prove that the group is exactly what you wrote. It is enough to show that, up to the action of an element of that group, the unique solution is $13.3.5.4.12.8.1.11.9.10.2.6$. First, one may assume of course $a=13$. Since the only way to decompose $8$ is $2+6$ and $3+5$, one might also assume, up to elements in the group, that $b=3$ and $c=5$. Clearly $\{l,k\}=\{2,6\}$.

An elementary calculation shows that the sums $b+f+j$ and $d+h+l$ equal $21$, so actually each of the $12$ vertices belongs to exactly two sets of $3$ vertices with sum $21$. Now, let's see where $12$ may be. Its complement to $21$ is $9$, which has $3$ writings : $1+8$, $4+5$ and $3+6$. The last one can not occur since $3$ and $6$ are already placed. It is then clear that one must have $d=4$ and $e=12$. From here it is easy to fill out the hexagone: Since $l+d+h=21$, $l$ can not be $2$, so $l=6$ and $h=11$. Further, $\{f,g\}=\{1,8\}$. If $f=1$, the sum $f+b+j$ can not be $21$. So $f=8$, $g=1$, and finally $i=9$.

It's not very elegant, but it works...

Here is the MAGMA code to generate your group:

G:=sub<Sym(12)|(1,3,5,7,9,11)*(2,4,6,8,10,12), (2,12)*(3,11)*(4,10)*(5,9)*(6,8), (2,3)*(5,6)*(8,9)*(11,12)*(4,10)>;

I have a small function written by Tim Dokchitser that recognises direct and semi-direct products of standard groups like cyclic, symmetric, dihedral, etc. The group you have described is isomorphic to $C_2\times {\mathfrak S}_4$. The unique normal subgroup of order 2 is generated by a reflection in the centre of the hexagon, so its non-trivial element is given by (a,g)(b,h)(c,i)(d,j)(e,k)(f,l).

The 4-cycles are somewhat harder to visualise. One of them is (a,d,g,j)(b,c,e,f)(h,i,k,l). The copy of ${\mathfrak S}_4$ that this is contained in (there are three normal subgroups isomorphic to ${\mathfrak S}_4$) is generated by the following elements:

• (b, l)(c, k)(d, j)(e, i)(f,h) (reflection in the axis through a and g)
• (a, i, e)(b, j, f)(c, k, g)(d, l, h) (counter-clockwise rotation by $2\pi/3$)
• (a, g)(b, e)(c, f)(d, j)(h, k)(i, l) (square of the above 4-cycle)
• (a, j)(b, k)(c, i)(d, g)(e, h)(f, l) ( a weird thing somewhat similar to the previous one)

If you have more questions about this group, ask away, since magma will tell me pretty much anything I want to know.