What's the deal with momentum in the infinite square well?

• Is the momentum operator $P=-i\hbar \frac{\mathrm d}{\mathrm dx}$ symmetric when restricted to the compact interval of the well? Are there any subtleties in its definition, via its domain or similar, that are not present for the real-line version?

(I henceforth assume the Hilbert space is $L^2([0,1],dx)$.)

It depends on the precise definition of the domain of $P$. A natural choice is $$D(P) = \{\psi \in C^2([0,1])\:|\: \psi(0)=\psi(1)=0\}\:.\tag{1}$$ With this definition $P$ is symmetric: (a) the domain is dense in $L^2([0,1], dx)$ and (b) the operator is Hermitian $$\langle P\psi| \phi \rangle = \langle \psi| P\phi \rangle\quad \mbox{for $\psi, \phi \in D(P)\:.$}\tag{2}$$ You can consider different definitions of the domain more or less equivalent. The point is that the self-adjoint extension are related with the closure of $P$ and not to $P$ itself, and you may have several possibilities to get the same closure stating from different domains. The situation is similar to what happens on the real line. There $P$ can be defined as a differential operator on $C_0^\infty(\mathbb R)$ or $\cal S(\mathbb R)$ (Schwartz' space), or $C^1_0(\mathbb R)$ and also interpreting the derivative in weak sense. In all cases the closure of $P$ is the same.

• Is the momentum operator $P=-i\hbar \frac{\mathrm d}{\mathrm dx}$ self-adjoint in these conditions? If not, why not, and what are the consequences in terms of the things we normally care about when doing one-dimensional QM?

It is not self-adjoint with the said choice of its domain (or with every trivial modification of that domain). The consequence is that it does not admit a spectral decomposition as it stands and therefore it is not an observable since there is no PVM associated with it.

Defining $P= -i \hbar \frac{\mathrm d}{\mathrm dx}$ on the real line with one of the domain said above, the same problem arises.

The general fact is that differential operators are never self-adjoint because the adjoint of a differential operator is not a differential operator since it cannot distinguish between smooth and non-smooth functions, because elements of $L^2$ are functions up to zero-measure sets. At most a symmetric differential operator can be essentially self-adjoint, i.e., it admits a unique self-adjoint extension (which coincides to the closure of the initial operator). This unique self-adjoint operator is the true observable of the theory.

• If it is not self-adjoint, does it admit a self-adjoint extension?

Yes. The canonical way is checking whether defect indices of $P$ with domain (1) are equal and they are. But the shortest way consists of invoking a theorem by von Neumann:

If a (densely defined) symmetric operator commutes with an antilinear operator $C$ defined on the whole Hilbert space and such that $CC=I$, then the operator admits self-adjoint extensions.

In this case $(C\psi)(x):= \overline{\psi(1-x)}$ satisfies the hypothesis.

If it does, is that extension unique?

NO it is not, the operator is not essentially self-adjoint.

If the extension is not unique, what are the different possible choices, and what are their differences? Do those differences carry physical meaning / associations / consequences? And what is a self-adjoint extension anyways, and where can I read up about them?

There is a class of self-adjoint extensions parametrized by elements $\chi$ of $U(1)$. These extensions are defined on the corresponding extension of the domain $$D_\chi(P) := \{\psi \in L^2([0,1],dx)\:|\:\psi' \mbox{exists a.e., is $L^2$ and $\psi(1) = \chi\psi(0)$} \}\:. $$ (It is possible to prove that with the said definition of $D_\chi$ the definition is consistent: $\psi$ is continuous so that $\psi(0)$ and $\psi(1)$ makes sense.) Next the self-adjoint extension of $P$ over $D_\chi(P)$ is again $-i\hbar \frac{d}{dx}$ where the derivative is interpreted in weak sense. The simplest case is $\chi=1$ and you have the standard momentum operator with periodic boundary conditions, that is self-adjoint. The other self-adjoint extensions are trivial changes of this definition. I do not know the physical meaning of these different choices (if any): the theory is too elementary at this stage to imagine some physical interpretation. Maybe with an improved model a physical interpretation arises.

• What is the spectrum and eigenvectors of the momentum operator and its extensions? How do they differ from each other? Is there such a thing as a momentum representation in this setting? If not, why not?

You can easily compute the spectrum which is a pure-point spectrum and the eigenvectors are shifted exponentials. If $\chi = e^{i \alpha}$ where $\alpha \in \mathbb R$, and we denote by $P_\alpha$ the associated self-adjoint extension of $P$ a set of eigenvectors is $$\psi^{(\alpha)}_n(x) = e^{i(\alpha + 2\pi n)x}$$ with eigenvalues $$p^{(\alpha)}_n := \hbar(\alpha + 2\pi n)\quad n \in \mathbb Z\::$$ The set of the $\psi^{(\alpha)}_n$ is a Hilbert basis because it is connected with the standard basis of exponentials by means of the unitary operator $(U_\alpha \psi)(x) = e^{i\alpha x} \psi(x)$. Essentially Nelson's theorem and the spectral decomposition theorem prove that $P_\alpha$ has pure point spectrum made of the reals $p_n^{(\alpha)}$. So a momentum representation exists as you can immediately prove.

• What is the relationship between the momentum operator (and its possible extensions) and the hamiltonian?

As you know, if you start form $H := -\hbar \frac{d^2}{dx^2}$ on $D(H):= \{\psi \in C^2([0,1]) \:|\: \psi(0)=\psi(1) =0\}$ (I assume $2m=1$) this is essentially self-adjoint, though the corresponding momentum operator with domain (1) is not. (The proof immediately arises from Nelson's theorem since $H$ is symmetric and admits a Hilbert basis of eigenfunctions.)

However there are also different candidates for the Hamiltonian operator arising by taking the second power of each self-adjoint extension of $P$ with domain (1). The spectrum is made of the second powers the elements of the spectrum of the corresponding seof-adjoint extension $\hbar^2(\alpha + 2\pi n)^2$

Do they commute? Do they share a basis?

Momentum and associated Hamiltonian commute and a common basis is that written above for the momentum.

Different self-adjoint extensions and different Hamiltonians do not commute as you easily prove by direct inspection.

• Do these problems have counterparts or explanations in classical mechanics? (nudge, nudge)

I do not know

And, more importantly: what are good, complete and readable references where one can go and get more information about this?

I do not know, many results are spread in the literature. It is difficult to collect all them. A good reference is Reed and Simon's textbook: Vol I and II.