How does this boundary term go to zero in the derivation of momentum (QM)?

In QM while doing these manipulations you always assume that the wave function is square integrable i.e. in $L^2(\mathbb R)$ space. Now mathematically speaking functions in the $L^2(\mathbb R)$ space can be wild. So rather than working with $L^2(\mathbb R)$ functions, we usually restrict ourselves to Schwartz functions $\mathcal S(\mathbb R)$, which are functions that satisfy

$$|f^{(n)} (x) \cdot x^m| < \infty \quad \text{for } x \to \infty \qquad \forall m,n $$

in other words Schwartz functions are functions, which fall faster than any power, regardless of how many derivatives you take. For example, $1/(1+x^2)$ on $\mathbb R$ is not a Schwartz function because

$$\frac{x^m}{1+x^2} \to \infty \quad \forall m>2$$

Note that $1/(1+x^2)$ is square integrable but not Schwartz. Another example would be $e^{-x^2}$ which you can show with a little bit of calculus is indeed a Schwartz function. Even though Schwarz functions look very restrictive, they are dense in $L^2$ norm, that is you can approximate the integral of any $L^2$ function with a sequence of Schwartz functions.

In physics we usually make the assumption that the wave function falls off "fast enough" and usually "fast enough" means being a Schwartz function. There is also an expectation that solutions to Schrödinger equation with nice enough potentials will be Schwartz functions. This is for example true for the harmonic oscillator or the hydrogen atom.

If we assume that the solutions are Schwartz functions then, you can easily see why you can drop terms like $x |\Psi|^2$ as $x \to \infty$.


A linear operator $A$ in a Hilbert space $\cal H$, is defined as a linear map associating elements of $\cal H$ to elements of $\cal H$.

The key information you need is that not every possible linear map having the whole $\cal H$ as domain is an operator if the Hilbert space is infinite dimensional (like $L^2(\mathbb R)$). There is a special set of linear operators (the bounded or continuous operators) mapping bounded sets into bounded sets, which can be defined over the whole Hilbert space, but other operators exist (the unbounded ones) whose domain is a proper subset of $\cal H$ and cannot be extended to the whole Hilbert space.

Position and momentum operators are important examples in QM of unbounded operators. In the case of the usual representation: $$ \begin{eqnarray} \hat x &:& \hat x \psi(x,t) \rightarrow x \psi(x,t) \\ \hat p &:& \hat p \psi(x,t) \rightarrow -i \hbar \frac{d}{dx}\psi(x,t). \end{eqnarray} $$ In the case of position, its domain $D_{\hat x}$ has to be the proper subset of the $L^2(\mathbb R)$ space whose elements are such that $x\psi(x,t) \in L^2(\mathbb R)$. In the case of momentum, $D_{\hat p}$ coincides with the subset of square integrable and differentiable functions.

Every time one is manipulating expressions containing unbounded operators, it is important to identify the underlying domain, if a precise mathematical meaning has to be given to the expressiones.

Coming to your expression, the starting point (expectation value of $\hat x$) requires that the wavefunction $\Psi(x,t)$ would be in $D_{\hat x}$. In order to exploit the fact that $\Psi(x,t)$ is solution of the time dependent Schrödinger equation, $\Psi(x,t)$ has also be in the domain $D_{\hat p^2}$, which is the subset of twice differentiable elements of $L^2(\mathbb R)$). Therefore, the whole formal manipulation implicitly assumes $\Psi(x,t) \in D_{\hat p^2} \cap D_{\hat x}$.

Vanishing of your expression is ensured in such a domain by Schwarz's inequality. Indeed, $$ \left| \int_{-\infty}^{+\infty} x\Psi^* \frac{d \Psi}{dx} dx \right| \le \sqrt{\int_{-\infty}^{+\infty} x^2\left|\Psi\right|^2 dx} \sqrt{\int_{-\infty}^{+\infty} \left|\frac{d \Psi}{dx} \right|^2 dx}. $$ Both the integrals on the right hand side of the previous inequality exist and are finite for any $\Psi \in D_{\hat p^2} \cap D_{\hat x}$. Therefore the integral on the left hand side is finite and consequently the integrand function must go to zero when approaching $-\infty$ and $+\infty$

Using dense sets of nice-behaving functions, like in Gonenc' answer is a different way to arrive to the same conclusion, although, in my opinion, it obfuscates the key role played by the unbounded operators which are at the starting point of your expression.