Short way for upper triangularization

Suppose we begin with the matrix $$A = \begin{bmatrix} 3 & 0 & 1 \\ -1 & 4 & -3 \\ -1 & 0 & 5 \\ \end{bmatrix}$$ and use "double Gaussian elimination". That is, whenever we multiply by an elementary row operation $E$ on the left, we must also multiply by the corresponding column operation $E^{-1}$ on the right. We begin by removing the $-1$ in the bottom left corner by doing a row operation of the form $R_3 \leftarrow R_3 + \lambda R_1$. We must then also perform the column operation $C_1 \leftarrow C_1 - \lambda C_3$. Since we want this to cancel the bottom-left cell, we see that $\lambda$ must satisfy $$-1 + 3 \lambda - \lambda(5 + \lambda) = 0 \iff \lambda^2+2\lambda+1 = (\lambda+1)^2 = 0$$ So we take $\lambda = -1$, and apply those operations to get $$A_1 = \begin{bmatrix} 4 & 0 & 1 \\ -4 & 4 & -3 \\ 0 & 0 & 4 \\ \end{bmatrix}$$

Now, this is almost in upper-triangular form: if we perform the paired operations $(R_1 \leftrightarrow R_2, C_1 \leftrightarrow C_2)$ then we end up with

$$A_2 = \begin{bmatrix} 4 & -4 & -3 \\ 0 & 4 & 1 \\ 0 & 0 & 4 \\ \end{bmatrix}$$

which is upper-triangular. Let $P$ be the matrix tracking the column operations, so that $$P = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 1 \\ \end{bmatrix}$$

Then we have $P^{-1} A P = A_2$.

$\newenvironment{gmatrix}{\left\lgroup\begin{matrix}}{\end{matrix}\right\rgroup}$Here it is a general way of finding Jordan normal forms. It applies to your question because all eigenvalues of $A$ are real.

Step 1: Computing the invariant factors of $λI - A$ to get its elementary divisors.

The monic greatest common divisor of all $1 × 1$ minors of $λI - A$ is $D_1(λ) = 1$, and that of all $2 × 2$ minors is $D_2(λ) = 1$, and that of the $3 × 3$ minors is$$ D_3(λ) = |λI - A| = (λ - 4)(λ - 4 + \sqrt{2})(λ - 4 - \sqrt{2}). $$ Thus the invariant factors of $λI - A$ are$$ d_1(λ) = D_1(λ) = 1,\quad d_2(λ) = \frac{D_1(λ)}{D_2(λ)} = 1,\\ d_3(λ) = \frac{D_3(λ)}{D_2(λ)} = (λ - 4)(λ - 4 + \sqrt{2})(λ - 4 - \sqrt{2}), $$ and the elementary divisors of $λI - A$ are$$ λ - 4,\ λ - 4 + \sqrt{2},\ λ - 4 - \sqrt{2}. $$

Step 2: Writing down the Jordan normal form of $A$ based on the elementary divisors of $λI - A$.

Since the elementary divisors of $λI - A$ are $λ - 4$, $λ - 4 + \sqrt{2}$, $λ - 4 - \sqrt{2}$, the Jordan normal form of $A$ is$$ J = \begin{gmatrix} 4 &&\\ & 4 - \sqrt{2} &\\ && 4 + \sqrt{2} \end{gmatrix}. $$

Step 3: Finding all the Jordan chains and the $P$.

In this case, it is easy to get that$$ x_1 = \begin{gmatrix} 0 \\ 1 \\ 0 \end{gmatrix}, x_2 = \begin{gmatrix} 1 + \sqrt{2} \\ 1 + 2\sqrt{2} \\ 1 \end{gmatrix}, x_3 = \begin{gmatrix} 1 - \sqrt{2} \\ 1 - 2\sqrt{2} \\ 1 \end{gmatrix} $$ are the three Jordan chains (here eigenvectors in fact) of $λ - 4$, $λ - 4 + \sqrt{2}$, $λ - 4 - \sqrt{2}$, respectively. Thus$$ P = (x_1, x_2, x_3) = \begin{gmatrix} 0 & 1 + \sqrt{2} & 1 - \sqrt{2}\\ 1 & 1 + 2\sqrt{2} & 1 - 2\sqrt{2}\\ 0 & 1 & 1 \end{gmatrix}. $$

There is a misprint in the announcement. The $(1,3)$ element of $A$ should be $+1$ rather than $-1$ in order for $A$ to be consistent with the calculations that follow in your posted question.

So let $A=\begin{bmatrix} 3&0&1\\-1&4&-3\\-1&0&5 \end{bmatrix}$ and set $N=A-4I= \begin{bmatrix} -1&0&1\\-1&0&-3\\-1&0&1 \end{bmatrix}$.

Then $N^2= \begin{bmatrix} 0&0&0\\4&0&-4\\0&0&0 \end{bmatrix}$ and $N^3=0$. So $N$ is nilpotent and $4$ is a triple eigenvalue of $A$ having a `Jordan chain' of length 3. A simple way to triangularize $A$ is then to calculate 3 vectors that generate this chain. For this you may note that $\ker N =\langle v_1\rangle $ is generated by $v_1=\begin{bmatrix} 0 & 1 & 0 \end{bmatrix}^T$ and $\ker N^2 =\langle v_1,v_2\rangle$ with e.g. $v_2=\begin{bmatrix} -1 & 1 & -1\end{bmatrix}^T$. Finally, set e.g. $v_3=\begin{bmatrix} 2 & 0 & -2\end{bmatrix}^T$ so that $\ker(N^3)={\Bbb R}^3=\langle v_1,v_2,v_3\rangle$.

Now set $R= [v_1 \; v_2\; v_3] = \begin{bmatrix} 0&-1&2\\1&1&0\\0&-1&-2\end{bmatrix}$ and we get: $A R = R \begin{bmatrix} 4 & 4 & 0\\ 0&4&4\\0&0&4\end{bmatrix}$ as wanted.

You may, of course, replace $v_2$ by $a v_2+b v_1$ with $a\neq 0$ which will change e.g. the $(1,2)$ element of the Jordan form (and similarly for $v_3$). My choice was just in order to get easy numbers and a nice end-result.