Is each squared finite group trivial?

It seems that every squared finite group is indeed trivial.

Let $G$ be a squared finite group with the subset $A$ showing the squared-ness of $G$. For any irreducible representation $\pi$ of $G$, denote $u_\pi = \sum_{g\in A} \pi(g) \in \operatorname{End} V_\pi$ where $V_\pi$ is the vector space of the representation. Then, by the condition on $A$, we have $u_\pi ^2 = \sum_{x,y\in A}\pi(xy) = \sum_{g\in G}\pi(g)$, which is $0$ if $\pi$ is not trivial, and $|G|$ if $\pi$ is trivial. That is, if $\pi$ is not trivial then $u_\pi$ is nilpotent, hence $\operatorname{tr}u_\pi=0$.

Expanding $\operatorname{tr} u_\pi$, we get (for nontrivial $\pi$) $$ 0=\operatorname{tr}u_\pi=\sum_{g\in A}\chi_\pi(g) = \sum_{g\in G}\frac{\left|g^G\cap A\right|\chi_\pi(g)}{\left|g^G\right|} $$ where $g^G$ is the conjugacy class of $g$ in $G$, and the last equation follows from the fact that $\chi_\pi$ is a class function (constant on each conjugacy class). As the set of characters is an orthogonal basis of the space of class functions, it follows that the function $g\mapsto \frac{\left|g^G\cap A\right|}{\left|g^G\right|}$ is proportional to $\chi_1$, that is independent on $g$. In particular we can substitute $1$ in $g$ and get $$ \frac{\left|g^G\cap A\right|}{\left|g^G\right|} = \frac{\left|\left\{1\right\}\cap A\right|}{1} $$ Therefore,

• if $1\in A$ then $g^G\subset A$ for any $g\in G$, which means that $A=G$.
• On the other hand, if $1\not\in A$, then $g^G\cap A=\emptyset$ for any $g\in G$, which means that $A=\emptyset$.

As it is clear that $|A|^2=|G|$, it follows that the only possible case is $G=A=\left\{1\right\}$.

I think, as implicitly suggested by Yemon Choi, it is possible to explain the proof of the answer of user49822 by making more use of idempotents. Suppose that the finite group $G$ is squared via the subset $A$. The element $ e = \frac{1}{|G|}\sum_{g \in G} g$ is a primitive idempotent of $\mathbb{C}G.$

Let $ f= \frac{1}{|A|}\sum_{a \in A} a.$ Then we have $f^{2} = ef = fe = e = e^{2}$. Thus $(f-e)^{2} = 0 = e(f-e) = (f-e)e.$ Now $f = e +(f-e)$ is the sum of commuting matrices (in the regular representation of $\mathbb{C}G$, say) with the second matrix nilpotent.

Thus $f$ has trace $1$ in the regular representation of $\mathbb{C}G.$ This forces $1 \in A$ since all non-identitiy elements of $G$ have trace zero in the regular representation. But then $A = \{1 \}$, since (as in Jeremy Rickard's comment) if $a \neq 1 \in A$, then $a = 1a = a1$ gives two different expressions for $a$. Alternatively, (using traces) the fact that $1$ appears with coefficient $|G|^{-1}$ in $e$ tells us that $1$ appears with multiplicity $|G|^{-1}$ in $f$ as well, so that $\sqrt{|G|} = |A| = |G|$ and $|G| = 1.$

Here is a proof for the abelian case, that perhaps has some chance to generalize.

Suppose $G$ is a squared finite group as witnessed by the subset $A$. Consider the element $$\alpha = \frac{1}{|A|} \sum_{a \in A} a$$ of the group algebra $\mathbb C G$. It is clear that $\alpha$ acts as the identity on the trivial representation. The squared condition implies that $$\alpha^2 = \frac{1}{|G|} \sum_{g \in G} g$$ which, as is standard, acts as the identity on the trivial rep and annihilates all other irreps. Thus $\alpha$ itself squares to zero on all nontrivial irreps.

In the abelian case where the irreps are 1-dimensional, this implies that $\alpha$ is just zero on the nontrivial irreps, so $\alpha = \alpha^2$ and $A = G$, which obviously only satisfies the bijectivity condition if $G$ is trivial. To extend this to the nonabelian case, one would have to rule out $\begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$ Jordan blocks.

(Note that I didn't quite use the full power of the squared condition, only that $A \times A \to G$ is $|A|^2/|G|$-to-one. In particular this proves that the only subset with this property in finite abelian groups is $A = G$. But that stronger statement may well fail in the nonabelian case, so perhaps the fact that $|A|^2 = |G|$ needs to be used in some essential way.)

Edit: As was pointed out in a (now deleted) comment, the abelian case is in fact rather trivial since obviously commutativity already tells you that if $|A| > 1$ the map can't be injective. Of course proving the abelian case was not really the main reason for posting this, but I think it's worth mentioning anyway!