Kalman decomposition using Hautus test

A.Γ. is right that you need to use the generalised eigenvectors, so do a Jordan decomposition instead of an eigendecomposition. So in case of the example in your question the $A$ matrix has the following generalised eigenvectors

$$ V = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{bmatrix} $$

where the Jordan canonical form of $A$ consists out of two blocks of size two and one, both with the eigenvalue one.

However $\text{rank}\left[A-\lambda\,I \quad B\right]$ and $\text{rank}\left[A^\top-\lambda\,I \quad C^\top\right]$ tells you nothing about which block and corresponding generalized eigenvectors are causing those matrices to lose rank. So this would be the limitation of using the Hautus test in order to obtain the Kalman decomposition. The structure of a Jordan block actually always ensures that the largest rank loss caused by that block is at most one. So all this method would tell you is how many blocks have rank loss, but not which and by how much. For example when

$$ A = \begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{bmatrix}, \quad B_1 = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \quad B_2 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$

where $A$ is one Jordan block of size three with eigenvalue zero, but for both $B_1$ and $B_2$ does $\left[A-\lambda\,I \quad B_i\right]$ lose rank. But for $B_1$ only one state is uncontrollable, while for $B_2$ two states are uncontrollable.

So this method fails when the matrix has an eigenvalue with algebraic multiplicity larger than one, but the rank loss (either for controllability, observability or both) is less then that. Because in that case it is not clear which subset of (generalised) eigenvectors needs to be taken.