Rigged Hilbert space and QM

I don't know of any books which use this language exclusively, but the basic idea is pretty straightforward:

All Hilbert spaces are isomorphic (if their dimensions match). This would present conceptual problems in quantum mechanics if we ever talked about the Hilbert space alone; how could we distinguish them? But it's OK because we are actually interested in a Hilbert space $\mathcal{H}$ equipped with an algebra of operators $\mathcal{A}$.

For example, the real difference between $\mathcal{H} = L^2(\mathbb{R})$ and $\mathcal{H} = L^2(\mathbb{R}^3)$: When we talk about the former, we're talking about $L^2(\mathbb{R})$ with the natural action of the 1d Heisenberg algebra $\mathcal{A}_1$ (generated by $P$ and $Q$ such that $[Q,P] = i\hbar$). When we talk about the latter, we're talking about the Hilbert space with the natural action of the 3d Heisenberg algebra $\mathcal{A}_3$.

Neither algebra actually acts on the entirety of $\mathcal{H}$. $(Q\psi)(x)= x\psi(x)$ doesn't necessarily lie in $L^2$. Likewise, the action of the differentiation operator $P = -i\hbar \frac{\partial}{\partial x}$ on a vector $v \in \mathcal{H}$ isn't defined if $v$ is not a differentiable function. And $P^2$ is only defined on twice-differentiable functions. However, there are some functions on which the action of any power $P^nQ^m$ is defined: If $v$ and all of its derivatives vanish faster at infinity than any polynomial, the action of any element of $\mathcal{A}_1$ is defined. Likewise, $\mathcal{A}_3$ really acts on the set $\mathcal{S}$ of functions in $L^2(\mathbb{R}^3)$ whose partial derivatives all vanish fast enough at infinity.

In general, if you have a Hilbert space and an algebra $\mathcal{A}$ of operators with continuous spectrum, there's a maximal subspace $\mathcal{S} \subset \mathcal{H}$ on which $\mathcal{A}$ acts. This is the subspace of $v \in \mathcal{H}$ for which $av$ is defined and $||a v|| < \infty$ for any $a \in \mathcal{A}$. It is called the space of smooth vectors for $\mathcal{A}$. (Exercise: $\mathcal{S}$ is dense in $\mathcal{H}$.)

$\mathcal{S}$ gets a topology from being a subspace of $\mathcal{H}$, but it actually has a much stronger topology from the family of seminorms $v \mapsto ||a v||$ (for $a \in \mathcal{A}$). This topology makes it a nuclear vector space.

Given $\mathcal{S} \subset \mathcal{H}$, you can construct the space $\mathcal{S}^* \supset \mathcal{H}$ of continuous (wrt the nuclear topology) complex-linear linear functionals on $\mathcal{S}$. (Here we are using the Riesz representation theorem to identify $\mathcal{H}$ with its dual $\mathcal{H}^*$.) This space should be thought of as the space of bras, in the Dirac bra-ket sense. The bra $\langle x |$ is the linear function which maps $\psi \in \mathcal{S}$ to $\psi(x) =\langle x | \psi \rangle $, aka, the Dirac delta function $\delta_x$ with support at $x$. (The space of kets is the conjugate space, consisting of conjugate-linear functionals on $\mathcal{S}$. The ket $|x \rangle$ maps a state $\psi \in \mathcal{S}$ to $\psi^*(x) = \langle \psi| x\rangle$.)

This space $\mathcal{S}$ is worth considering because it gives rigorous meaning to the idea that elements of $\mathcal{A}$ with continuous spectrum have eigenvectors, and that you can expand some states in these eigenbases. The elements of the algebra $\mathcal{A}$ can't have eigenvectors in $\mathcal{H}$ if they have continuous spectrum. But they do have eigenvectors in the space of bras. The definition is a standard extension-by-duality trick: $v \in \mathcal{S}^*$ is an eigenvector of $a \in \mathcal{A}$ with eigenvalue $\lambda$ if $(av)(\psi) = \lambda v(a^*\psi)$ for all $\psi \in \mathcal{S}$. (Exercise: $\langle x|$ is the eigenbra with eigenvalues $x$ of the position operator $Q$.)

The triplet $(\mathcal{S}, \mathcal{H}, \mathcal{S}^*)$ is a rigged Hilbert space. The language of rigged Hilbert spaces was invented to capture the ideas I've outlined above: the smooth vectors of an algebra of operators with continuous spectrum, and the dual vector space where the eigenbases of these operators live. The language actually matches the physics very nicely -- especially the bra-ket formalism -- but it provides a level of precision that's not really necessary for most calculations (e.g., with floating point arithmetic).

There aren't too many examples of rigged Hilbert spaces treated at length in the literature. I plan to write an article on the rigged Hilbert space approach to the Schroedinger hydrogen atom. You can use the PhD thesis by Rafael de la Madrid (http://galaxy.cs.lamar.edu/~rafaelm/dissertation.html) and all the articles here: https://scholar.google.com/citations?user=OqTexTYAAAAJ&hl=en