Find real number $a$ such that matrix $A$ is NOT diagonalisable
(These Linear Algebra course notes by Michael Stoll, 2007 provide good background to my following assertions. As does the Wikipedia page on characteristic polynomials.)
Use the following facts:
a) A matrix $M \in \mathrm{Mat}(n,F)$ (where $n$ is the size of the matrix and $F$ is some field) is diagonalisable if and only if its minimal polynomial $m_M(x)$ is a product of distinct monic linear factors.
b) The characteristic polynomial of a matrix $M$ is defined as $p_M(x)=\mathrm{det}(xI-M)$
c) Matrices satisfy their own characteristic equations: $p_M(M)=0$ for all $M$. Hence a matrix's minimal polynomial always divides its characteristic polynomial: $m_M(x) \, \vert \, p_M(x)$.
d) The discriminant of a cubic polynomial $ax^3+bx^2+cx+d$ is given by: $$ \Delta_3 = b^2c^2-4ac^3-4b^3d-27a^2d^2+18abcd$$ If $\Delta_3>0$ then the equation as three distinct real roots. If $\Delta_3=0$ then the equation has a repeated root and all its roots are real. If $\Delta_3<0$ then the equation has just one real root.
For your matrix,
$$ A:= \left( \begin{matrix}2 & a & -1\\0 & 2 & 1\\-1 & 8 & -1\end{matrix} \right) $$
we have that
$$ \begin{align} p_A(x) &= \mathrm{det} \left( \begin{matrix} x-2 & -a & 1\\0 & x-2 & -1\\1 & -8 & x+1\end{matrix} \right) \\ \\ &= (x-2)((x-2)(x+1)-8) + (a - (x-2)) \\ &= x^3 -3x^2-9x+22+a \end{align} $$
With some calculation, the discriminant of $p_A$ is equal to $$\Delta_3 = -27(a-5)(a+27)$$
Firstly, $$\Delta_3>0 \iff -27<a<5$$ In this case, $p_A$ has three distinct real roots then it will factorise into three distinct linear factors. Hence $m_A(x)$ will also factorise into distinct linear factors and $A$ will be diagonalisable.
So $-27<a<5$ implies that $A$ is diagonalisable.
Secondly, $$\Delta_3=0 \iff a \in \{-27,5\}$$
If $a=-27$ then $$ A:= \left( \begin{matrix}2 & a & -1\\0 & 2 & 1\\-1 & 8 & -1\end{matrix} \right) $$ and this matrix can be checked to have a maximum of two linearly independent eigenvectors $$ v_1:= \left( \begin{matrix}-10\\1\\3\end{matrix} \right) \qquad v_2:= \left( \begin{matrix}-8\\-1\\3\end{matrix} \right)$$ and so $A$ is not diagonalisable. (We could also consider the Jordan Form of $A$ as calculated by @WillJagy in their answer to this question.)
Similarly for $a=5$, $A$ turns out to not be diagonalisable.
Thirdly,
$$\Delta_3<0 \iff a \in (-\infty , -27) \cup (5, \infty )$$
In this case $p_A(x)=(x- \lambda )(x^2+ \beta x + \gamma )$ where $\lambda$ , $\beta$ and $\gamma$ are real and $(x^2+ \beta x + \gamma )$ cannot be factorised over the reals. Since $m_A$ divides $p_A$ we have three options: $$ \begin{align} \mathrm{(i)} \qquad & m_A(x)=(x- \lambda ) \\\mathrm{(ii)} \qquad & m_A(x)=(x^2+ \beta x + \gamma ) \\\mathrm{(iii)} \qquad & m_A(x)=(x- \lambda )(x^2+ \beta x + \gamma ) \end{align}$$
It is obvious that $(A-\lambda I )$ is not equal to $0$ for any $\lambda$ so we can rule out option $\mathrm{(i)}$. But in both of the options $\mathrm{(ii)}$ and $\mathrm{(iii)}$, $m_A$ is not a product of distinct monic linear factors and hence $A$ is not diagonalisable.
Therefore:
$A$ is not diagonalisable over $\mathbb{R}$ if and only if $a \in (-\infty , -27] \cup [5, \infty )$
Edit: diagonalisability over $\mathbb{C}$
Even if working in $\mathbb{C}$, the cases $\Delta_3>0$ and $\Delta_3=0$ above remain exactly the same: in these cases the characteristic polynomial has no complex roots.
However, $\Delta_3<0$ now gives us that $p_A$ can be factorised into three distinct linear factors in $\mathbb{C}[x]$. Hence $m_A(x)$ will also factorise into distinct linear factors in $\mathbb{C}[x]$ and $A$ will be diagonalisable.
Hence $A$ is not diagonalisable over $\mathbb{C}$ if and only if $a \in \{-27,5\}$.
In general, for $A \in \mathrm{Mat}(n,\mathbb{C})$, if the discriminant of $p_A$ is not equal to $0$ then $p_A$ has no repeated roots and $A$ will be diagonalisable over $\mathbb{C}$. If the discriminant of $p_A$ is equal to $0$ then check the eigenvectors of $A$ (or form the Jordan Form of $A$) in order to check individual cases of diagonalisability.
Let's consider such an $a$. Then $A$ must necessarily have a repeated eigenvalue, otherwise it would be diagonalizable. If $f(t) = \det(tI-A)$ is the characteristic polynomial, then $f$ and $f'$ must have a common root. This severely restricts the values that $a$ can take.
Can you continue from here? Note that above is a necessary condition - after you figure out all potential candidates for $a$, you need to check if it actually makes $A$ non-diagionalizable.
Addendum. I have been slow on one point; if $a$ lies outside a certain range, then there must be complex eigenvalues, that is, not real numbers. That is what the other answers and comments are emphasizing, that it might not be possible to diagonalize over the real numbers.
Another way for non-diagonal Jordan form is real but repeated (generalized) eigenvalues, where there is not a basis of eigenvectors. This is the aspect I address below.
ORIGINAL: The derivative of the characteristic polynomial is $3(x-3)(x+1).$ When $a=5,$ we do get the characteristic polynomial as a multiple of $(x-3)^2.$ When $a=-27,$ we get the characteristic polynomial as a multiple of $(x+1)^2.$ These lead to non-diagonal (but real) Jordan forms, see below.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
With $a=5$ and $$ \left( \begin{array}{ccc} 1&-9&5 \\ 1&27&5 \\ -6&18&6 \\ \end{array} \right) \left( \begin{array}{ccc} 2&4&5 \\ -1&1&0 \\ 5&1&1 \\ \end{array} \right) = \left( \begin{array}{ccc} 36&0&0 \\ 0&36&0 \\ 0&0&36 \\ \end{array} \right) $$ we get
$$ \frac{1}{36} \; \left( \begin{array}{ccc} 1&-9&5 \\ 1&27&5 \\ -6&18&6 \\ \end{array} \right) \left( \begin{array}{ccc} 2&5&-1 \\ 0&2&1 \\ -1&0&-1 \\ \end{array} \right) \left( \begin{array}{ccc} 2&4&5 \\ -1&1&0 \\ 5&1&1 \\ \end{array} \right) = \left( \begin{array}{ccc} -3&0&0 \\ 0&3&1 \\ 0&0&3 \\ \end{array} \right) $$ The final version is called Jordan Normal Form. In the U.S. it is taught with the nilpotent part above the diagonal. In some countries, for example Uruguay, it is taught with any extra $1$s below the diagonal.
Actually using the Jordan form for something merely requires writing out $$ \frac{1}{36} \; \left( \begin{array}{ccc} 2&4&5 \\ -1&1&0 \\ 5&1&1 \\ \end{array} \right) \left( \begin{array}{ccc} -3&0&0 \\ 0&3&1 \\ 0&0&3 \\ \end{array} \right) \left( \begin{array}{ccc} 1&-9&5 \\ 1&27&5 \\ -6&18&6 \\ \end{array} \right) = \left( \begin{array}{ccc} 2&5&-1 \\ 0&2&1 \\ -1&0&-1 \\ \end{array} \right) $$
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
WHEN $a=-27,$ With $$ \left( \begin{array}{ccc} 1&-17&-3 \\ 1&19&-3 \\ 6&6&18 \\ \end{array} \right) \left( \begin{array}{ccc} 10&8&3 \\ -1&1&0 \\ -3&-3&1 \\ \end{array} \right) = \left( \begin{array}{ccc} 36&0&0 \\ 0&36&0 \\ 0&0&36 \\ \end{array} \right) $$ we get
$$ \frac{1}{36} \; \left( \begin{array}{ccc} 1&-17&-3 \\ 1&19&-3 \\ 6&6&18 \\ \end{array} \right) \left( \begin{array}{ccc} 2&-27&-1 \\ 0&2&1 \\ -1&0&-1 \\ \end{array} \right) \left( \begin{array}{ccc} 10&8&3 \\ -1&1&0 \\ -3&-3&1 \\ \end{array} \right) = \left( \begin{array}{ccc} 5&0&0 \\ 0&-1&1 \\ 0&0&-1 \\ \end{array} \right) $$
$$ \frac{1}{36} \; \left( \begin{array}{ccc} 10&8&3 \\ -1&1&0 \\ -3&-3&1 \\ \end{array} \right) \left( \begin{array}{ccc} 5&0&0 \\ 0&-1&1 \\ 0&0&-1 \\ \end{array} \right) \left( \begin{array}{ccc} 1&-17&-3 \\ 1&19&-3 \\ 6&6&18 \\ \end{array} \right) = \left( \begin{array}{ccc} 2&-27&-1 \\ 0&2&1 \\ -1&0&-1 \\ \end{array} \right) $$
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=