Can I break the degeneracy of energy eigenstates if I know which irrep of the group they transform as?

  1. It is not possible to “break the degeneracy” by combining degenerate states: any unitary transformation inside the degenerate subspace will produce a different set of eigenstates of $H$, but they will all have the same eigenvalues.

  2. This does not prevent you from organizing the states in your (degenerate) subspace into irreps of some group. Presumably the elements of this group commute with the Hamiltonian so you will get legitimate eigenstates which also carry group labels.

  3. The reason you might want to use the group is that some perturbation might lift the degeneracy so that states in different irreps have different eigenvalues (or at least some irreps have different eigenvalues, as there is to guarantee that all degeneracies are lifted).

  4. In the example you give, you start with $\phi_0(x)\phi_1(y)$ and $\phi_0(y)\phi_1(x)$. Imagine you were to add an interaction between the two particles - say some sort of potential which would be of the type $\kappa (x-y)^2$. A perturbative treatment of this interaction would depend, to first order, on the average of $(x-y)^2$. This perturbation is invariant under $S_2$, the group of permutation of the two coordinates. This group has 2 1-dimensional representation and (no surprise) the irreps are spanned (up to normalization) by \begin{align} \Psi_{\pm}(x,y)&=\phi_0(x)\phi_1(y)\pm \phi_1(x)\phi_0(y)\, ,\\ &\sim e^{-\lambda(x^2+y^2)}(y\pm x) \end{align} The states $\Psi_{\pm}(x,y)$ are still degenerate under the original 2D h.o. oscillator Hamiltonian but, in a perturbative treatment, the effect of interaction in $(x-y)^2$ would depend, to first order, on the average of $(x-y)^2$ evaluated with either $\Psi_{\pm}(x,y)$, and this average is different for the two combinations. Moreover, it is easy to check using parity that \begin{align} \int dx\,dy\, \Psi_-(x,y)(x-y)^2\Psi_+(x,y)=0\, . \end{align} Thus, by organizing your state according to irreps of $S_2$, which leaves your perturbation invariant, the degeneracy is lifted by the perturbation and the basis states for the irreps of $S_2$ are eigenstates of the Hamiltonian plus perturbation, at least to first order.

To understand the splitting of terms there is a massive paper (translated in English) by Hans Bethe

Bethe, Hans A. "Splitting of terms in crystals." Selected Works Of Hans A Bethe: (With Commentary). 1997. 1-72.

(originally Ann.Physics 3 p.133 (1929))

and the other canonical source is

Tinkham, M., 2003. Group theory and quantum mechanics. Courier Corporation.