Probability of commutation in a compact group
Edit: let me reformulate my answer since I apparently didn't answer the right question.
(a) if $K$ is a compact Lie group, the commuting probability is positive iff $K_0$ is abelian. As you noticed, $\Leftarrow$ is trivial.
Conversely, assume that $K_0$ is not abelian. We can view $K$ as Zariski-closed in matrix group, all its (Lie) components are also Zariski-closed, and are the irreducible components. Hence, if by contradiction the set of commuting pairs has positive measure (or equivalently has nonempty interior, or equivalently has dimension $2\dim K$), then it contains a product of two cosets: $aK_0\times bK_0$ for some $a,b\in K$. So $agbh=bhag$ for all $g,h\in K_0$.
Putting $g,h=1$, we get $ab=ba$. Putting $h=1$, we get $agb=bag=1$ for all $h$ which using $ab=ba$ yields $gb=bg$ for all $g\in K_0$. Putting $g=1$, we get $abh=bha$ for all $h$, which using $ab=ba$ yields $ah=ha$ for all $h\in K_0$. Thus $a,b$ commute and centralize $K_0$. So the formula simplifies as $hg=gh$ for all $g,h\in K_0$.
(b) let $K$ be a compact connected group with positive commuting probability. The property passes to Lie quotients of $K$, which are therefore abelian (the argument in the connected Lie case is contained in the above: the set of commuting pairs is a proper subset). Since $K$ is projective limit of its Lie quotients, it follows that $K$ is abelian.
If $K/K_0$ is finite, we reach the conclusion that $K_0$ is abelian.
Here is a proof that avoids Lie theory. Suppose $G$ is a compact connected group in which the probability that two elements commute is $p>0$. We claim $G$ is abelian.
Step 1 is to prove that the elements of $G$ have boundedly many conjugates. Let $X_n$ be the set of elements of $G$ with at most $n$ conjugates. Clearly $X_n$ is closed, and $$ p = \int 1/|x^G| d\mu(x) \leq 1/n + \mu(X_n),$$ so $\mu(X_n) > 0$ provided $n > 1/p$. This implies that $X_n$ generates an open subgroup, which must be $G$ since $G$ is connected.
Step 2: $[G,G]$ is finite. In general a theorem of Bernhard Neumann asserts that if the elements of a group have boundedly many conjugates then $[G,G]$ is finite.
Step 3 is to look at the commutator map $[,]: G\times G \to [G,G]$. The codomain is discrete, and $[G,1]=1$, so there is some neighbourhood of the identity $U$ such that $[G,U]=1$. This $U$ generates an open central subgroup of $G$, which again must be $G$ since $G$ is connected.
Admittedly, not obviously simpler than YCor's proof. (All this and a little more is written up at https://randompermutations.com/2015/02/06/commuting-probability-of-compact-groups/.)
Edit, after reading @NateEldredge's answer:
Actually, this is quite easy. We just need this, which is very easily proved:
Lemma: Let $G$ be a compact group and $H\leq G$ a closed subgroup. Then the following are equivalent:
- $\mu(H)>0$;
- $H$ has finite index;
- $H$ is open.
In particular, if $G$ is connected then $\mu(H) > 0$ iff $H=G$.
Suppose $G$ is compact and connected. Let $x \in G$. Then the centralizer $C_x$ is a closed subgroup of $G$, so by the lemma $\mu(C_x) > 0$ iff $C_x = G$, i.e., iff $x \in Z(G)$. Therefore we have $$ P(xyx^{-1}y^{-1} = 1) = \int \mu(C_x) \,d\mu(x) = \int 1_{x \in Z(G)} \, d\mu(x) = [G:Z(G)]. $$ Now $Z(G)$ is also closed, so by the lemma again if the LHS is positive we must have $Z(G)=G$.
If the compact connected group $K$ is second-countable (hence metrizable, hence Polish), one can prove it with pretty much straight measure theory. (I am not sure if this argument remains valid for non-metrizable groups.)
It's based on the following elementary lemma which is due (in some form) to Steinhaus:
Lemma. Let $G$ be a locally compact Polish group with a left-invariant Haar measure $\mu$, and $A \subset G$ be a Borel set with $\mu(A) > 0$. Then the set $A^{-1} A$ contains an open neighborhood of the identity. In particular, if $A$ is a subgroup of $G$ then $A$ is open.
You can find it as Exercise 17.13 (ii) in Kechris, Classical Descriptive Set Theory. You begin by showing that the function $x \mapsto \mu(xA \triangle A)$ is continuous, which uses the regularity of the measure or else Lusin's theorem. (The proof I know uses dominated convergence and hence only shows that this function is sequentially continuous, which is why I am unsure about the non-metrizable case.)
So let $E = \{(x,y) : xy = yx\} \subset K \times K$. This set is closed, and in particular, measurable. The probability of two elements commuting is $p = (\mu \times \mu)(E)$ where $\mu$ is the left Haar probability measure on $K$. Let $C_x$ denote the centralizer of $x$, which is closed. We have $(x,y) \in E$ iff $y \in C_x$, and thus by Fubini's theorem $$p = \iint 1_E(x,y)\,\mu(dy)\,\mu(dx) = \iint 1_{C_x}(y)\,\mu(dy)\,\mu(dx) = \int \mu(C_x) \,\mu(dx).$$
If we suppose $p >0$, then the set $B = \{x : \mu(C_x) > 0\}$ must have positive measure. But if $\mu(C_x) > 0$, then by our lemma, $C_x$ is an open subgroup of $K$. It was already closed, and $K$ is connected, so $C_x = K$, i.e. $x$ is in the center $Z$ of $K$. Thus $B \subset Z$, so $Z$ is a closed subgroup with positive measure. Applying our lemma again, $Z$ is open and so $Z=K$, i.e. $K$ is abelian.
There is also be a "Baire category" analogue of this statement in the Polish case: if $E$ is nonmeager in $K \times K$, then $K$ is abelian. You could prove it by a nearly identical argument: use the Pettis lemma (Kechris 9.9) in place of Steinhaus above, and the Kuratowski-Ulam Theorem (Kechris 8.41) in place of Fubini. (Though maybe there's a simpler argument, since a nonmeager closed set has to have nonempty interior.)