Dimensionality in quantum mechanics

To be quite frank, while the textbook by Griffiths may be a decent introduction to quantum mechanics, you should not rely on it for anything mathematics related.

That hermitian matrices can be orthogonally diagonalized is not a 'rather technical fact'. It is a completely standard result (known to both mathematicians and non-mathematicians), which is covered in any introductory linear algebra course. It finds application in a wide range of subjects, not just quantum mechanics.

There is a vast mathematical literature devoted to making mathematical sense of quantum mechanical concepts. Saying that this massive field, which lies in the intersection between physics and mathematics, is simply concerned with hermitian matrices, and then stating that the results do not carry over to the infinite dimensional setting... I don't know what to say. At the very least, it is an extreme misrepresentation of the actual state of affairs.

When should you expect to have a complete set of eigenvectors of a Hamiltonian? Usually, this happens if you consider a confining potential. Examples include the particle in a box, like you mentioned, and the harmonic oscillator.

When should you not expect to have a complete set of eigenvectors? Usually, this happens if your system can 'break apart', becoming essentially a non-interacting free system. Examples include the free particle.

When should you expect to have eigenvectors, but not a complete set? If your system has bound states, but it is also possible to break it apart. Examples include the hydrogen atom, with bound states below the ionization threshold.

Since you mention chemistry, it is in order to mention that for many applications, in practice one often considers only the space spanned by the bound states. In this subspace, the eigenvectors do of course constitute a complete set. This is what is usually done in introductory quantum mechanics courses, where one calculates the energy levels of the hydrogen atom. Often, it is not even mentioned that the spectrum of the hydrogen atom contains the positive real line along with these negative eigenvalues! On the other hand, this fact is not really important if you only care about the bound states.

If you are interested in the mathematics of quantum mechanics, one of the key topics is spectral theory of unbounded (self-adjoint) operators. The classics 'Perturbation Theory for Linear Operators' by Kato and 'Methods of Modern Mathematical Physics' by Reed and Simon are recommendable. For more recent texts, I would also recommend 'Unbounded Self-adjoint Operators in Hilbert Space' by Schmüdgen and 'Quantum Mathematical Physics' by Thirring. Of course, all of these texts require a firm background in mathematics, with a focus on functional analysis.

Finally there are a bunch of questions on this site which are related to your question:

When Schrodinger operator has discrete spectrum?

why is the spectrum of the schrödinger operator discrete?

Spectrum of Laplace operator with potential acting on $L^2(\mathbb R)$ is discrete

When the point spectrum is discrete?

Proving Compactness of Resolvent Of an Operator

Operators with compact resolvent

Selfadjointness of the Dirac operator on the infinite-dimensional Hilbert space

Hermitian and self-adjoint operators on infinite-dimensional Hilbert spaces

Physicists persist in writing things that do not make sense mathematically, but there is a mathematically rigorous version of it: the Spectral Theorem for densely defined self-adjoint operators on Hilbert space. Most functional analysis texts will cover it.

Turning my comments into an answer and adding details since this is an important question in mathematical physics:

Many potentials will cause serious issues regarding completeness of eigenfunctions of Schrodinger operators. Really, we don't need completeness of the eigenfunctions, though it is desirable. That said, we can guarantee that many physical systems have nice spectra and this is usually all we really care about for most systems.

In general, constructing self-adjoint operators is difficult because you need to get the domain exactly right. We often settle for essentially self-adjoint operators. These operators are very close to self-adjoint but they just don't have quite the full domain or range (but are pretty close!). This is good enough for physicists (and often mathematicians because many theorems work out the way you want). Particularly you can use the spectral theorem for essentially self-adjoint operators. Simply having a symmetric operator (one where $\langle Ax,y\rangle = \langle x,Ay\rangle$ for $x,y$ in the domain of $A$) is not nearly enough to guarantee (essential) self-adjointness. You need to know some extra things to guarantee (essential) self-adjointness. I can elaborate on this if desired.

With that said, there are conditions that can be placed on the quantum mechanical potentials that can guarantee essential self-adjointness. Many quantum mechanical potentials fall into this regime - NOT ALL. Moreover, just because a potential does not fall into this regime, it does not necessarily mean that it does not give rise to a complete set of eigenfunctions. I'm pulling this from Galindo and Pascual's Quantum Mechanics Volume $1$. They state that:

Let the potential $U$ be piecewise continuous, then define $U_{\pm} = \lim_{x\to\pm\infty} U(x)$. If one of $U_{\pm}$ is finite and decays faster than $x^{-1}$ to the limiting value, if $U_{\pm} = +\infty$, or if $U_{\pm} = -\infty$ but $U(x) \ge -ax^2 - b$ for some $a,b> 0$, then the corresponding Schrodinger equation is essentially self-adjoint on $C_0^{\infty}(\Bbb R)$.

In these cases, the spectrum is real. Particularly, if eigenvalues exist, they are real. This last bit is not guaranteed and requires some work to show for various systems. Consider $U \equiv 0$ on $\Bbb R$, then there are no eigenvalues (since the generalized eigenfunctions are complex exponentials which are not $L^2$-normalizable).