How to compute the similarity transformation matrix
In order to find your $P$, you can do as follows:
- First, you find a diagonal matrix $D$ to which both $A$ and $B$ are equivalent. For this, you need to find the eigenvalues of both matrices and if they coincide, they are equivalent. (The general case would be more involved: but in yours, both matrices diagonalize easily.)
- Then you have to find bases of eigenvectors for both matrices and form with them change of bases matrices $S$ and $T$ such that
$$ D = S^{-1}A S \qquad \text{and} \qquad D = T^{-1}BT \ . $$
- Now you'll have
$$ S^{-1}A S = T^{-1}BT \qquad \text{and hence} \qquad AST^{-1} = ST^{-1}B \ . $$
So $ST^{-1}$ will be your matrix $P$.