confused about a matrix problem
Suppose the ground field is real or complex. We are going to show that $B$ must share a common eigenvector with $A$. Note that $B$ must be nonsingular, otherwise $k=\frac23$ but $(AB)^5\ne A^5B^5$.
By Cayley-Hamilton theorem, $(AB)^5=pAB+qI$ and $B^5=rB+sI$ for some scalars $p,q,r,s$. Since $(AB)^5$ and $B^5$ are nonsingular, $(p,q),(r,s)\ne(0,0)$. The equation in question can be rewritten as $pAB+qI=A^5(rB+sI)$, or equivalently, $$ (pA-rA^5)B = sA^5-qI.\tag{1} $$ Since the two eigenvalues $\frac12(5\pm\sqrt{33})$ of $A$ have different magnitudes, the powers of $A$ cannot possibly be scalar matrices. It follows that both sides of $(1)$ are nonzero.
If $pA-rA^5$ is nonsingular, then $B=(pA-rA^5)^{-1}(sA^5-qI)$ commutes with $A$ and hence the two matrices share a common eigenvector.
If $pA-rA^5$ is singular, then both $pA-rA^5$ and $sA^5-qI$ are rank-one matrices. Let $pA-rA^5=xy^T$ for some nonzero vectors $x$ and $y$ and let $u,v$ be two eigenvectors of $A$ corresponding to the two different eigenvalues of $A$ respectively. Then $(1)$ implies that $$ xy^TBu=au\ \text{ and }\ xy^TBv=bv\tag{2} $$ for some scalars $a$ and $b$. As $xy^T=pA-rA^5$ is a polynomial in $A$, we also have $$ xy^Tu=cu\ \text{ and }\ xy^Tv=dv\tag{3} $$ for some scalars $c$ and $d$. Since $y\ne0$ and $u,v$ are linearly independent, $y^Tu$ and $y^Tv$ cannot be both zero. Assume that $y^Tu\ne0$. Then the first equation in $(3)$ implies that $c\ne0$ and $x$ is a nonzero scalar multiple of $u$. The second equation in $(3)$ thus implies that $d=0$ and $\operatorname{span}(v)=\ker(y^T)$. But then the second equation in $(2)$ implies that $b=y^TBv=0$, i.e. $Bv\in\ker(y^T)=\operatorname{span}(v)$. Hence $Bv$ is a scalar multiple of $v$.
In other words, $A$ and $B$ must share a common eigenvector. Since the two linearly independent eigenvectors of $A$ (up to scaling) are $(-3\pm\sqrt{33},\,6)^T$, we must have $$ \pmatrix{k&2\\ 3&9}\pmatrix{x\\ 6}\propto\pmatrix{x\\ 6}, $$ for some $x\in\{-3+\sqrt{33},\ -3-\sqrt{33}\}$. Thus $$ \frac{kx+12}{3x+54}=\frac{x}{6}. $$ Since $x^2+6x=24$ when $x\in\{-3+\sqrt{33},\ -3-\sqrt{33}\}$, the above equation implies that $$ k=\frac1x\left[\frac{x(3x+54)}{6}-12\right] =\frac1x\left(\frac{x^2+18x}{2}-12\right) =\frac1x\left(\frac{12x+24}{2}-12\right) =6. $$ Therefore the only possible solution is $k=6$, and it is indeed a solution because $B=A+5I$ in this case. The cases are different over other ground fields. E.g. over $GF(2)$ or $GF(3)$, every $k$ in the field is a solution.
Well, $A$ is invertible, and $B$ is not invertible if and only if $k=\frac23$, and in this case it is easy but slightly laborious to check $(AB)^5\neq A^5B^5$.
So $A,B$ are invertible, and we reduce to 4 quartics in $(BA)^4=A^4B^4$. By repeated squaring and multiplying $$ A^4B^4= \begin{bmatrix} 199&290\\435&634 \end{bmatrix} \begin{bmatrix} 522 + 108 k + 18 k^2 + k^4 & 2 (837 + 93 k + 9 k^2 + k^3)\\3 (837 + 93 k + 9 k^2 + k^3)& 8055 + 108 k + 6 k^2\end{bmatrix} = \begin{bmatrix} 199 k^4 + 870 k^3 + 11412 k^2 + 102402 k + 832068 & 2 (199 k^3 + 2661 k^2 + 34167 k + 1334538)\\ 3 (145 k^4 + 634 k^3 + 8316 k^2 + 74622 k + 606348)&6 (145 k^3 + 1939 k^2 + 24897 k + 972510) \end{bmatrix} $$ and $$ (BA)^4=\begin{bmatrix} k+6 & 2k+8\\30&42 \end{bmatrix}^4 =\begin{bmatrix} k^4 + 204 k^3 + 11736 k^2 + 201024 k + 629136 & 2 (k^4 + 184 k^3 + 9336 k^2 + 143904 k + 437760)\\ 30 (k^3 + 180 k^2 + 8616 k + 109440) & 12 (5 k^3 + 800 k^2 + 33480 k + 380748) \end{bmatrix} $$ So we want to solve $$ \begin{bmatrix} 199 k^4 + 870 k^3 + 11412 k^2 + 102402 k + 832068 & 2 (199 k^3 + 2661 k^2 + 34167 k + 1334538)\\ 3 (145 k^4 + 634 k^3 + 8316 k^2 + 74622 k + 606348)&6 (145 k^3 + 1939 k^2 + 24897 k + 972510) \end{bmatrix} = \begin{bmatrix} k^4 + 204 k^3 + 11736 k^2 + 201024 k + 629136 & 2 (k^4 + 184 k^3 + 9336 k^2 + 143904 k + 437760)\\ 30 (k^3 + 180 k^2 + 8616 k + 109440) & 12 (5 k^3 + 800 k^2 + 33480 k + 380748) \end{bmatrix} $$ The most promising is the right-hand column because the lower-right is just a cubic and the coefficients is more manageable, yielding: $$ \begin{cases} k^4 - 15 k^3 + 6675 k^2 + 109737 k - 896778 = 0\\ 45 k^3 + 113 k^2 - 14021 k + 70338 = 0 \end{cases} $$ Taking out a copy of $(k-6)$, the resulting polynomials in $k$ $$ k^3 - 9 k^2 + 6621 k + 149463, 45 k^2 + 383 k - 11723 $$ are coprime. So $k=6$ is the unique solution.