Is the Perron-Frobenius dimension of a G-Set given by its cardinality?

For $H \subset G$, write $e_H$ for the basis element of $B(G)$ corresponding to the $G$-set $G/H$. Recall that for $K \subset G$, $$ e_H\cdot e_K = \sum\limits_{H g K \in H \backslash G / K} e_{H \cap g K g^{-1}}. $$

Say that $K$ is subconjugate to $H$ if $K$ is conjugate to a subgroup of $H$. This gives rise to a partial order on the set of conjugacy classes of subgroups of $G$, and hence on the basis elements $e_H$. With respect to this partial ordering, $M_{e_H}$ is lower-triangular for each $H \subset G$. The eigenvalues are then the diagonal elements. The diagonal element corresponding to $e_{\{\textrm{id}_G\}}$ is $|G/H|$. The diagonal element corresponding to $e_K$ for $K \subset G$ is bounded above by the number of $(H, K)$-double cosets in $G$, which is bounded above by $|G/H|$.

Lets fix an orbit $X = G/H$. It suffices to determine the asymptotic growth of the trace of multiplication by $X^n$, since the maximal positive eigenvalue(s) dominates the sum $\sum_i \lambda_i^n$.

This trace is the sum $$\sum_{K \subset G} \langle G/K, X^n \times G/K \rangle,$$ where the sum is over conjugacy classes of subgroups, and the bracket counts the number of disjoint orbits of type $G/K$.

Notice that the term $\langle G, X^n \times G \rangle$ equals $|X|^n$ (one free orbit for each $x^n \times e$). Conversely, $\langle G/K, X^n \times G/K \rangle$ must have size $\leq |X|^n |G/K|/|G/K| = |X|^n$.

Thus the exponent of the order growth is $|X|$, and so $|X|$ is the maximal eigenvalue. The class $[G]$ is an eigenvector corresponding to this eigenvalue.

Edit: As Darij points out, in this case Perron--Frobenius only guarantees that $$ max_{\lambda \in {\rm eig}(A)} |\lambda|$$ can be achieved by some positive real $\lambda$. There may be other complex eigenvalues of the same absolute value. In this case more argument is required to ensure that there is not cancellation.

Edit2: Here is a strategy to resolve the issue. Let $z_1, \dots, z_k$ be the eigenvalues achieving the maximum. Assume we can show that there exists an eplison such that $$S = \{n \in \mathbb N ~|~ Re(z_i^n) \geq -\lambda + \epsilon ~ \forall i \}$$ is infinite. Then there would be an infinite subset $S \subset \mathbb N$ and a $C \in \mathbb R_{> 1}$ such that $C \lambda^s \geq Tr(A^s) \geq 1/C \lambda^s$ for all $s \in S$.

In our case, we have $$|X|^n \leq {\rm Tr}(A^n) \leq \#\{\text{ subgroups }\}|X|^n$$ for all $n$. Restricting to the infinite subset $S$ we would see that $\lambda = |X|$.