How is it possible a quadratic Hermitian Hamiltonian $H = b^\dagger b^\dagger + b b$, with $b$ boson, cannot be diagonalized?

Essentially, you are trying to prove that your initial Hamiltonian can be rewritten as $$H = r a^\dagger a + sI\tag{1}$$ where $r$ and $s$ are reals and $a^\dagger$ and $a$ satisfy bosonic ccrs and there is a vacuum vector for $a$.

This is a much stronger condition than diagonalizability!

However, it is possible to prove that the requirement above cannot be satisfied. In fact, it immediately arises from (1) that $H$ is either bounded below or above depending on the sign of $r$, because

$$\langle \psi| H \psi \rangle = r \langle a \psi| a \psi \rangle + s \langle \psi |\psi \rangle = r ||a\psi||^2 + s||\psi||^2$$ Instead (with some technical hypothesis on the domain and some other technicality as Stone-von Neumann's theorem) one can prove that $b^\dagger b^\dagger + bb$ is not bounded below nor above. Therefore operators $a$ and $a^\dagger$ cannot exist.

To prove that the initial Hamiltonian cannot be bounded, observe that defining $X = b + b^\dagger$ and $P = i(b-b^\dagger)$, these operators satisfy standard ccr up to a real factor and $H$ is proportional to $P^2-X^2$ which is unbouded below and above (here technicalities should be used). In fact, this is the Hamiltonian of a repulsive oscillator so it is unbouded below, but a unitary transformation which therefore preserves the spectrum swaps $X$ and $P$ changing the sign of $H$, hence $H$ is also unbounded above.

Finally, all that does not mean that $H$ is not "diagonalizable". $H$ is at least Hermitian, presumably selfadjoint on some domain (it depends on details) thus it admits a PVM over the reals. However, the spectrum is continuos as it happens for the repulsive oscillator, instead of discrete.