What determines the maximal dimension of the irreps of a (finite) group?

A simple bound on the largest dimension of a complex irreducible representation (which is either equal to or half of the largest dimension of a real irreducible representation) is the following: we know that

• $|G| = \sum d_i^2$ where $d_i$ are the dimensions of the irreducibles,
• the number of (complex) irreducibles is the number $c(G)$ of conjugacy classes, and
• the size $a(G) = |G^{ab}|$ of the abelianization is the number of $1$-dimensional irreducibles (so the number of $d_i$ terms equal to $1$).

It follows that the largest dimension $d_{max}$ satisfies $a(G) + d_{max}^2 \le |G| \le a(G) + (c(G) - a(G)) d_{max}^2$, and rearranging these gives

$$\sqrt{ \frac{|G| - a(G)}{c(G) - a(G)} } \le d_{max} \le \sqrt{|G| - a(G)}.$$

$c(G)$ is a measure of "how abelian" $G$ is; it's a nice exercise to show that $\frac{c(G)}{|G|}$ is the probability that two random elements of $G$ commute. Roughly speaking this means that $d_{max}$ is a measure of "how nonabelian" $G$ is. For example, if $G = A_5$ is the icosahedral group then $|G| = 60, a(G) = 1, c(G) = 5$ gives

$$ \sqrt{ \frac{59}{4} } \approx 3.84 \dots \le d_{max} \le \sqrt{59} \approx 7.68 \dots $$

so $4 \le d_{max} \le 7$, and since we also know that the dimensions $d_i$ divide $|G|$ we have $4 \le d_{max} \le 6$, and the true value $d_{max} = 5$ is right in the middle. Loosely speaking this says that $A_5$ is "more nonabelian" than, say, a dihedral group, which satisfies $d_{max} = 2$.

This bound is most useful when the abelianization is large. A different bound useful when the center $Z$ is large is the following: we know that

• by Schur's lemma every irreducible representation has a central character, and if $\lambda : Z \to \mathbb{C}^{\times}$ is a central character, then the irreducibles with central character $\lambda$ can be identified with simple modules over the twisted group algebra obtained by quotienting $\mathbb{C}[G]$ by the relations $z = \lambda(z)$ for $z \in Z(G)$,
• every twisted group algebra as above has dimension $|G/Z|$, so the dimensions $d_i(\lambda)$ of the irreducibles with central character $\lambda$ satisfy $|G/Z| = \sum d_i(\lambda)^2$,
• the number of irreducibles with a fixed central character is the number of conjugacy classes of $G/Z$ satisfying a certain condition, and in particular is at most the number of conjugacy classes of $G/Z$.

Now it follows that the largest dimension $d_{max}$ satisfies $d_{max}^2 \le |G/Z| \le c(G/Z) d_{max}^2$, which gives

$$\sqrt{ \frac{|G/Z|}{c(G/Z)} } \le d_{max} \le \sqrt{|G/Z|}.$$

For example, the upper bound is tight for a finite Heisenberg group $H_3(\mathbb{F}_p)$, which satisfies $|G/Z| = p^2$ and has $p^2$ one-dimensional characters and $p - 1$ irreducibles of dimension $p$. The lower bound actually produces $1$ here which shows that it can be worse than the previous lower bound (which applied here gives $\sqrt{ \frac{p^3 - p^2}{p^2 + p - 1} } \approx \sqrt{p}$). The size of the center is another measure of "how abelian" $G$ is so this gives another sense in which $d_{max}$ measures "how nonabelian" $G$ is.

Your question touches on many issues in group representation theory, and I can only give a few general remarks which may point you in interesting directions for further reading.

As to your question regarding the maximal real irreducible representation of a finite group, there is an interesting connection with the Frobenius Schur indicator.

If $\chi$ is a (complex) irreducible character of a finite group $G$, the Frobenius-Schur indicator of $\chi$ is denoted by $\nu(\chi)$ defined to be $0$ if $\chi$ is not real-valued, to be $-1$ if $\chi$ is real-valued, but $\chi$ may NOT be afforded by a representation over $\mathbb{R}$, and to be $1$ if $\chi$ is afforded by a representation over $\mathbb{R}.$ For example, the unique irreducible complex character of degree $2$ of the quaternion group of order $8$ has Frobenius-Schur indicator $-1$, and the unique irreducible character of degree $2$ of the dihedral group of order $8$ ( I mean the one with $8$ elements) has Frobenius-Schur indicator $1$.

The number of solutions of $x^{2}=1 $ in the finite group $G$ is equal to $\sum_{\chi} \nu(\chi) \chi(1)$, where $\chi$ runs over the complex irreducible characters of $G$.

This is especially useful if all irreducible characters $\chi$ of $G$ have $\nu(\chi) = 1$, which is always the case for $G = S_{n}$ (the symmetric group).

The FS-indicator may (in principle at least) be calculated via the formula $\nu(\chi) = \frac{1}{|G|} \sum_{g \in G} \chi(g^{2}).$

In the case of the alternating group of degree $5$, for example, all irreducible characters $\chi$ have $\nu(\chi) = 1$, the irreducible characters have degree $1,3,3,4,5$. Hence we get $\sum_{\chi} \nu(\chi)\chi(1) = 16$, and there are indeed $16$ solutions of $x^{2} = 1$ in $G$ (the identity and fifteen elements of order $2$).

As to the question of what you term degeneracy, there is some ambiguity (related to the Frobenius-Schur indicator and also the Schur index). For example, the quaternion group of order $8$ has a $4$-dimensional representation which is irreducible as a real representation, but which is equivalent to the sum of two equivalent $2$-dimensional complex irreducible representations. An absolutely irreducible real representation is a real irreducible representation which remains irreducible as a complex representation. This is a representation whose character $\chi$ is irreducible as a complex character and has $\nu(\chi) = 1.$

A real irreducible representation which is not absolutely irreducible is one which is not irreducible as a complex representation. Such a representation may afford a character of the form $2\chi$ where $\chi$ is a complex irreducible character with $\nu(\chi) = -1$, or it may afford a character of the form $\chi + \overline{\chi}$, where $\chi$ is a complex irreducible character with $\nu(\chi) = 0$ (ie $\chi$ is not real-valued).

In terms of complex irreducible representations, it is one of the earliest theorems in group theory (due to C. Jordan) that if a finite group $G$ has a complex representation of degree $n$ (irreducible or not), then $G$ has an Abelian normal subgroup whose index is bounded in terms of $n$. This also applies to real irreducible representations.

If we restrict to complex irreducible representations which are primitive (that is, can not be induced from a representation of a proper subgroup), this tells us that if $G$ has a primitive complex irreducible representation of degree $n$, then the number of possibilities for $G/Z(G)$ is bounded in terms of $n$.

On the other hand, the symmetric group $S_{n+1}$ always has a real irreducible representation of degree $n$, and has order $(n+1)!$, yet has no non-identity Abelian normal subgroup if $n >3.$ This is related to the "generic" worst case bound for Jordan's Theorem, and is genuinely an upper bound for that Theorem if $n$ is large enough.

I think that in general, it is very difficult to relate the order of generators of a finite group $G$ with the largest degree of its real (or complex) irreducible representations. For example, there are arbitrarily large finite simple groups $G$ which may be generated by an element of order $2$ and an element of order $3$, and there is therefore no upper bound on the dimensions of the real irreducible representations of finte groups which may be generated by an element of order $2$ and an element of order $3$.

Later edit: Another general fact which is often useful, is a result of N. Ito, which states that if the finite group $G$ has an Abelian normal subgroup $A$, then the degree of any complex irreducible representation of $G$ is a divisor of the index $[G:A].$