How to diagonalize this matrix...

First step: Find the eigenvalues of your matrix.

Eigenvectors are vectors $\mathbf x$ such that upon being multiplied by a matrix $A$, they are only scaled by a number. That is $A\mathbf x = \lambda \mathbf x$, where $\lambda$ is just a number, called the eigenvalue associated with the eigenvector $\mathbf x$.

The way to do this is to subtract the $\lambda \mathbf x$ from both sides to get $A\mathbf x -\lambda \mathbf x=\mathbf 0$. Now factor out the $\mathbf x$ to get $(A-\lambda I)\mathbf x = \mathbf 0$, where $I$ is the identity matrix -- note: we need the identity matrix because adding a matrix and a scalar is undefined.

This equation, $(A-\lambda I)\mathbf x = \mathbf 0$ has a nontrivial solution (a solution where $\mathbf x \ne 0$) if and only if $\det(A-\lambda I)=0$ (can you prove this?).

So let's look at that determinant: $$\det(A-\lambda I) = \det\left(\begin{bmatrix} 1 & 2 & 0 \\ 2 & 1 & 0 \\ 0 & 0 & -3 \\ \end{bmatrix} - \begin{bmatrix} \lambda & 0 & 0 \\ 0 & \lambda & 0 \\ 0 & 0 & \lambda \end{bmatrix}\right)$$ $$= \det\left(\begin{bmatrix} 1 - \lambda & 2 & 0 \\ 2 & 1 - \lambda & 0 \\ 0 & 0 & -3-\lambda \\ \end{bmatrix}\right) = 0$$

Using our formula for the determinant we see that this is equal to $(1-\lambda)((1-\lambda)(-3-\lambda)-0)-2(2(-3-\lambda)-0)=0$. This has solutions (if I didn't make a mistake) of $\lambda =-3$ or $\lambda = -1$ or $\lambda = 3$.

These are the eigenvalues of this matrix.

Second step: Put these numbers into the diagonal of a matrix -- with zeroes everywhere else -- in whichever order you like and you're done! Yippee!

$$\begin{bmatrix} -3 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 3\end{bmatrix}$$ is one of your possible diagonal matrices.

---

Let's verify our solution, though.

Third step: Find the eigenvectors associated with the eigenvalues of this matrix.

Let's start with the first eigenvalue $\lambda = -3$ and plug it into our matrix $A-\lambda I$ and then try to find all of the solutions to $(A-\lambda I)\mathbf x = \mathbf 0$.

$$(A-\lambda I)\mathbf x = \begin{bmatrix} 4 & 2 & 0 \\ 2 & 4 & 0 \\ 0 & 0 & 0 \end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$$

Now we note that if $A\mathbf x= \mathbf 0$, then $R\mathbf x = \mathbf 0$ where $R$ is the RREF form of $A$. I assume you know how to row reduce a matrix, so I'll just show you the $R\mathbf x=\mathbf 0$ form:

$$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0\end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0\end{bmatrix}$$

Here we can clearly see that all solutions $\mathbf x$ to this equation are of the form $t\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$, for some scalar $t$. Thus $\begin{bmatrix} 0 \\ 0 \\ 1\end{bmatrix}$ is the eigenvector associated with the eigenvalue $\lambda =-3$ of the matrix $A$.

Following the exact same procedure: you see that the other two eigenvectors are $\mathbf x_{\lambda_2} = \begin{bmatrix} -1 \\ 1 \\ 0\end{bmatrix}$ and $\mathbf x_{\lambda_{3}} = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}$.

Fourth step: Verify your diagonal matrix by constructing a block matrix $P$ of your eigenvectors and using the equation $P^{-1}AP=\Lambda$.

Stick your eigenvectors into the columns of a matrix: $$P = \begin{bmatrix} 0 & -1 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 0\end{bmatrix}$$

Find the inverse $P^{-1}$ of that matrix: $$P^{-1} = \begin{bmatrix}0 & 0 & 1 \\ -\frac 12 & \frac 12 & 0 \\ \frac 12 & \frac 12 & 0\end{bmatrix}$$

Now multiply out the expression $P^{-1}AP$ and see if you get back your diagonal matrix.

$$P^{-1}AP = \begin{bmatrix}0 & 0 & 1 \\ -\frac 12 & \frac 12 & 0 \\ \frac 12 & \frac 12 & 0\end{bmatrix}\begin{bmatrix} 1 & 2 & 0 \\ 2 & 1 & 0 \\ 0 & 0 & -3 \\ \end{bmatrix}\begin{bmatrix} 0 & -1 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 0\end{bmatrix} = \begin{bmatrix} -3 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 3\end{bmatrix}$$

Success!

Since Bye_World has done everything, I just want to try proving his statement:

This equation, $(A−\lambda I)x=0$ has a nontrivial solution (a solution where $x\neq 0$) if and only if $\det(A−\lambda I)=0$"

We first assume that $B = A−\lambda I$ is invertible (or $\det(B)≠0$). If this is true, the equation can be rewritten as:

$$B^{-1}Bx = B^{-1}0=0$$

Since $B^{-1}B=I$ (the identify matrix), then $x=0$, which is false. Then B is not invertible or $\det(B)=0$.