In what way are the Hilbert spaces of non-interacting quantum field theories different from each other?

However, having consulted a few different sources I have been repeatedly told that the state space for, say, a quantum scalar field is not the same as that of, say, the quantised Dirac field.

The problem here is the meaning of "the same". When we talk about state spaces being "the same", we do not mean that they are isomorphic as Hilbert spaces. The space $L^2(\mathbb{R})$ is isomorphic as a Hilbert space to the space $L^2(\mathbb{R}^3)$, yet we clearly recognize one of them as the state space of a scalar particle in one dimension and the other as the state space of a scalar particle in three dimensions.

This is because the state space isn't really about choosing the Hilbert space - it's about choosing a representation of the algebra of observables on it. The space $L^2(\mathbb{R})$ is "the" state space for a particle in 1 dimension because the Stone-von Neumann theorem tells us that every (nice) representation of the 1d canonical commutation relations $[x,p] = \mathrm{i}$ is isomorphic to the one on $L^2(\mathbb{R})$ where $x$ is represented by multiplication and $p$ is represented by differentiation. The same holds for the space $L^2(\mathbb{R}^3)$ and the 3d commutation relations $[x_i, p_j] = \mathrm{i}\delta_{ij}$.

Similarly, when we say that the state spaces of a scalar particle and a spinor are "not the same", we also mean that the representation of an observable - namely spin - is different. For a spinor there is a non-trivial representation of an internal $\mathfrak{su}(2)$ commuting with $x$ and $p$, for a scalar there isn't, there's just the angular spin operator $x\times p$.

So you see there is a lot of additional information attached to saying some "space" is a "space of states" that is rarely made explicit (because usually the representations that are attached are "obvious", for varying values of "obvious"...). It's not about the spaces themselves, it's about representations.

While all (infinite-dimensional, separable) Hilbert spaces are isomorphic, this doesn't mean that they are necessarily a good fit for the theory. Theorems assure us that there is an isomorphism between the Hilbert space of scalar fields and Dirac fields, but this does not mean that our theory will look "natural".

To avoid unnecessary complications, let's consider a simple case, of non-relativistic quantum mechanics for scalars and spinors (this will more or less correspond to the one particle Hilbert spaces in QFT, up to a change of the symmetry groups). You could also do the QFT case where the Hilbert space is wavefunctionals on a function space, $L^2(D(\mathbb{R}^3), \mathcal{D}\phi)$, but that will not fundamentally change the point we're making here.

We have the two Hilbert spaces $\mathcal{H}_0$ and $\mathcal{H}_{1/2}$, where

\begin{eqnarray} \mathcal{H}_0 &=& L^2(\mathbb{R}^3)\\ \mathcal{H}_{1/2} &=& L^2(\mathbb{R}^3) \otimes V_{1/2}\\ \end{eqnarray}

The natural Hilbert space for a (non-relativistic) spinor is the Hilbert space composed of the product of the usual Hilbert space with $V_{1/2}$, the irreducible projective representation of $\mathrm{SO}(3)$. This is a Hilbert space which carries a non-trivial representation of the rotation group.

In other words, a spinor wavefunction will look something like

\begin{eqnarray} \psi(\vec{x}) = \xi(\vec{x}) \begin{pmatrix}\psi^+\\\psi^-\end{pmatrix} \end{eqnarray}

This is basically the kind of thing you'd see for, say, the solution of the Pauli equation for a hydrogen atom. The inner product of this Hilbert space is simply enough the inner product of the usual Hilbert space and the spinor space :

\begin{eqnarray} \langle \psi_1, \psi_2 \rangle = \int \xi_1^*(x) \xi_2(x) \left[ \psi^+_1 \psi^-_2 + \psi^-_1 \psi^+_2 \right] dx \end{eqnarray}

This makes it both an inner product and invariant under spinor rotation of our wavefunction.

What is stopping us from using the usual Hilbert space for this particle? Let's see what happens if we do.

The exact theorem tells us that any two Hilbert spaces with bases of the same cardinality are isomorphic by a unitary transformation that maps an orthonormal basis of one space to the other. Let's pick some orthonormal basis of $\mathcal{H}_0$. The exact form doesn't quite matter, but we'll pick one indexed by integers rather than the weird momentum basis, so this could be Hermite polynomials for instance.

Any wavefunction of this Hilbert space is therefore expressible as

\begin{eqnarray} \psi(x) = \sum_{n = 0}^\infty a_n \psi_n(x) \end{eqnarray}

and more to the point, it maps it to the Hilbert space $\ell^2(\mathbb{N})$.

The Hilbert space $\mathcal{H}_{1/2}$ has basically the same basis, except that it is of course the tensor product basis $\psi_n \otimes e_i$, for $i = 1, 2$. So our wavefunctions will be

\begin{eqnarray} \psi(x) = \sum_{i = 1}^2 \sum_{n = 0}^\infty a_{i, n} \psi_n(x) \otimes e_i \end{eqnarray}

Our basis has dimension $2 \aleph_0$ rather than $\aleph_0$, so we are still firmly in the same cardinality, and we can also map it to $\ell^2(\mathbb{N})$.

From there, it's not terribly complicated to find an isomorphism of those two Hilbert spaces, by simply picking any bijection between two such copies of $\mathbb{N}$. This is the trivial case of mapping, say, integers to even integers, so that we could map, for instance, $\psi_{2n}$ to $\psi_{n} \otimes e_1$ and $\psi_{2n+1}$ to $\psi_n \otimes e_2$. There's no lack of such bijections.

Then there is a very simple isomorphism of $\mathcal{H}_0$ to $\mathcal{H}_{1/2}$, and every wavefunction will have a corresponding wavefunction in the other, with the appropriate eigenvalues given the properly changed operators.

However, what we just did was pure nonsense. We're sending eigenvalues of some operator and associating it with the spin for absolutely no reason. The translated operators, for say, rotation or spin, would bear absolutely no resemblance to what they usually are, and most likely would be a garbled mess. If you'd pick, say, the spin operator $S_z$ in our basis,

\begin{eqnarray} S_z &=& | + \rangle \langle + | - | - \rangle \langle - | \end{eqnarray}

or, to use the complete basis,

\begin{eqnarray} S_z &=& (\sum_{n = 0}^\infty |\psi_n \rangle \langle \psi_n|) \otimes \left(| + \rangle \langle + | - | - \rangle \langle - | \right) \end{eqnarray}

Then we are somewhat mapping this to

\begin{eqnarray} S_z &\to& (\sum_{n = 0}^\infty |\psi_{2n} \rangle \langle \psi_{2n}| - |\psi_{2n + 1} \rangle \langle \psi_{2n + 1}|) \end{eqnarray}

While this will 100% give you the appropriate solutions, given the proper isomorphisms between the states, operators, inner product and symmetries, there is little point to this. The information of what we're doing is completely obscured by this isomorphism, and it is also completely arbitrary here since we could have picked both any basis for $\mathcal{H}_0$ or any bijection between $\mathbb{N}$.

The same reasoning applies for QFT, except more complex since we are dealing with either the Fock space or functional Hilbert spaces, but the basic idea remains, since those are still infinite dimensional separable Hilbert spaces.