How do I tell if matrices are similar?

There is something called "canonical forms" for a matrix; they are special forms for a matrix that can be obtained intrinsically from the matrix, and which will allow you to easily compare two matrices of the same size to see if they are similar or not. They are indeed based on eigenvalues and eigenvectors.

At this point, without the necessary machinery having been covered, the answer is that it is difficult to know if the two matrices are the same or not. The simplest test you can make is to see whether their characteristic polynomials are the same. This is necessary, but not sufficient for similarity (it is related to having the same eigenvalues).

Once you have learned about canonical forms, one can use either the Jordan canonical form (if the characteristic polynomial splits) or the rational canonical form (if the characteristic polynomial does not split) to compare the two matrices. They will be similar if and only if their rational forms are equal (up to some easily spotted differences; exactly analogous to the fact that two diagonal matrices are the same if they have the same diagonal entries, though the entries don't have to appear in the same order in both matrices).

The reduced row echelon form and the reduced column echelon form will not help you, because any two invertible matrices have the same forms (the identity), but need not have the same determinant (so they will not be similar).

My lecturer, Dr. Miryam Rossett, provided the following in her supplementary notes to her linear 1 course ( with a few small additions of my own ):

1. Show that the matrices represent the same linear transformation according to different bases. This is generally hard to do.
2. If one is diagonalizable and the other not, then they are not similar.
3. Examine the properties of similar matrices. Do they have the same rank, the same trace, the same determinant, the same eigenvalues, the same characteristic polynomial. If any of these are different then the matrices are not similar.
4. Check the geometric multiplicity of each eigenvalue. If the matrices are similar they must match. Another way of looking at this is that for each $\lambda_i$, $\dim \ker(\lambda_i I-A_k)$ must be equal for each matrix. This also implies that for each $\lambda_i$, $\dim \text{im}(\lambda_i I-A_k)$ must be equal since $\dim \text{im}+\dim \ker=\dim V$
5. Assuming they're both diagonalizable, if they both have the same eigenvalues then they're similar because similarity is transitive. They'e diagonalizable if the geometric multiplicities of the eigenvalues add up to $\dim V$, or if all the eigenvalues are distinct, or if they have $\dim V$ linearly independent eigenvectors.

Numbers 3 and 4 are necessary but not sufficient for similarity.

First off, the $2\times2$ case is easy: two distinct $2\times2$ matrices are similar if and only if they have the same characteristic polynomial, and neither of them is a scalar matrix (multiple of the identity). This is essentially because for such small matrices the characteristic polynomial leaves very little room for variation of the minimal polynomial (and other invariant factors that in this case follow from it).

There is a decision method that works in full generality (any field$~K$, any size$~n$ square matrices) and is theoretically simple: compute the invariant factors by determining the Smith normal form of the matrices $XI_n-A$ and $XI_n-B$ (those whose determinants define the respective characteristic polynomials) as matrices with polynomials in$~X$ as entries; then $A$ and $B$ are similar if and only if identical lists of invariant factors are found.

With "theoretically simple" I mean the method only uses basic operations in$~K$ you might be expected to perform: arithmetic of scalars and testing for equality; notably it does not require finding roots or factoring polynomials as any method based on eigenvalues would do. Essentially the Smith normal form algorithm is a generalisation of Gaussian elimination, but working over a PID rather than over a field (which here means: doing polynomial operations on polynomial entries, rather than just using scalars), and allowing column operations as well as row operations. The final goal is to reduce the matrix to its Smith normal form, which has nonzero (polynomial) entries only on the main diagonal, and which are monic polynomials such that each one divides the next (in general there could also be trailing zeros, but that does not happen here since our initial matrices have monic, therefore nonzero, determinants). The diagonal entries found that are different from$~1$ are the invariant factors of $A,B$ respectively; the last one is the minimal polynomial and their product is the characteristic polynomial. Those two polynomials can of course also be computed by other methods, and if either of them differs between $A$ and $B$ this suffices to show they are not similar; in the (rather rare) general case however all invariant factors are needed. In most cases there will be many diagonal entries$~1$, followed by one or very few non-constant polynomials that are the invariant factors.

Being theoretically simple does not mean the procedure is easy to perform (by hand), as it is like in iterated version of the Euclidean algorithm in the ring $K[X]$ of polynomials, which algorithm is notable for producing very complicated intermediate results, at least when $K=\Bbb Q$ (and over $\Bbb R$ or $\Bbb C$ the algorithm isn't really effective, as testing for equality is not accurately possible). It works fine over finite fields though.