Efficient computation of conjugacy classes of a small group.

From the defining relations try to get a description of all the elements.

Any element is a "word" involving $a$ and $b$.

Last condition $b^{-1}ab = a^2$ can be rewritten as $ab= ba^2$. This modified version says whenever a comes ahead of b it can be pushed to come behind b, but as $a^2$. So all words in a and b can be reformulated to have b's in the front and a's at end. As $a^9=b^6=1$, any element can be written as $b^m a^n$ with $m=0,1,2,\ldots, 8, \ n=0,1,2,\ldots, 5$, giving 54 possibilities.

Now try to describe conjuates of elements of type $a^k$, then $b^k$ and then $a^k b^l$ by any $g$ ($g$ has to be of the form $b^m a^n$ as above). It is doable.

$G = \{a^mb^n:0\le m\le 8,0\le n\le 5\}$. For an element $a^mb^n\in G$, $b^{-1}a^mb^nb = a^{2m}b^n$ and thus for a fixed $n$, $a^mb^n$ (with $m = 1,2,4,5,7,8$ or with $m = 3,6$) are in a same conjugacy class. Moreover, $$a^{-1}a^mb^na = a^{m-1}b^nab^{-n}b^n = a^{m-1}a^{5^n}b^n$$since $ba^2b^{-1} = a = (a^2)^5$ and $a^2$ is also a generator of $\langle a\rangle$. Note that a conjugate by $a$ or $b$ does not change $n$. Therefore, we only need to discuss $n$.

(1) $n=0$, then $a^{-1}a^mb^na = a^mb^n$, so the conjugate by $a$ is fixed. So $\{1\},\{a,a^2,a^4,a^5,a^7,a^8\},\{a^3,a^6\}$ are conjugacy classes.

(2) $n=1$, then $a^{-1}a^mb^na = a^{m+4}b^n$. So $\{b,ab,\dots, a^8b\}$ is a conjugacy class, since $m+4$ runs all the $m$'s.

(3) $n=2$, then $a^{-1}a^mb^na = a^{m+24}b^n = a^{m+6}b^n$. Now $a^3b^2$, $a^6b^2$ and $b^2$ are in the same conjugacy class, while other $m$ forms another conjugacy class.

(4) $n=3$, then $a^{-1}a^mb^na = a^{m+124}b^n = a^{m+7}b^n$. The same as (2) since it runs all the $m$'s.

(5) $n=4$, then $a^{-1}a^mb^na = a^{m+3}b^n $. The same as (3).

(6) $n=5$, then $a^{-1}a^mb^na = a^{m+1}b^n$. The same as (2).