Why can't a particle penetrate an infinite potential barrier?

Imagine a finite potential well of the form

$$ V(x) = \begin{cases} 0 & |x| < L/2 \\ V_0 & {\rm otherwise}\end{cases} $$

You can solve Schrodinger's equation in the usual way, by splitting the domain in three parts, the resulting wave function will look something like this

$$ \psi(x) = \begin{cases} \psi_1(x) & x < L/2 \\ \psi_2(x) & |x| \leq L/2 \\ \psi_3(x) & x > L/2\end{cases} $$

Inside the box $\psi_2(x) \sim e^{\pm ikx}$, but outside the box you will find

$$ \psi_3(x) \sim e^{-\alpha x} $$


$$ \alpha = \frac{\sqrt{2m(V_0 - E)}}{\hbar} $$

Now calculate the limit $V_0\to\infty$ (infinity potential barrier), and you will see that $\psi_3(x)\to 0$, same as $\psi_1(x)$. So in that sense the particle cannot penetrate the barrier and remains confined in the region $|x| \leq L/2$

The relation between the particle's wave function $\psi(x)$, potential $V(x)$ and energy is $$ E = \int dx\ \psi^*(x)\left(-\frac{\hbar^2}{2m}\psi''(x) + V(x)\psi(x)\right) \quad \label((*) $$ Suppose $V(x)$ is bounded from below and is equal to $+\infty$ on some interval $[x_1,x_2]$. If $\psi(x)\neq 0$ for $x\in[x_1,x_2]$, then the energy $E$ is infinite. The term containing second derivative is always non-negative, so it can not compensate this infinity.

Update. This relation is well known in the quantum mechanics. I didn't mention that the norm of a wave function is usually taken to be $1$: $$ \int dx\ \psi^*(x)\psi(x) = 1 $$ Under this condition the Schrodinger equation $$ -\frac{\hbar^2}{2m}\psi''(x) + V(x)\psi(x) = E\psi(x) $$ been multiplied by $\psi^*(x)$ and integrated by $x$ gives the relation (*).

The term $$ -\frac{\hbar^2}{2m}\int dx\ \psi^*(x)\psi''(x) $$ corresponds to the kinetic energy of a particle, so it must be non-negative. Indeed, integration by parts leads to the following manifestly non-negative expression $$ \frac{\hbar^2}{2m}\int dx\ \psi'^*(x)\psi'(x). $$ By the way, quantity $\psi''(x)/\psi(x)$ can be either positive or negative.

Gec's answer is the one I would consider rigorous, but the intuitive answer is this:

Suppose you put a particle detector in the barrier. How often do you expect to measure a particle there? Answer: never, because if you did this then the particle after measurement would be in a position eigenstate that would force you to conclude it had infinite (expectation value of) energy. And we're disallowing that.

The only states where there is no chance to ever measure the particle in the barrier are those with $\psi=0$ inside the barrier (or at least over a dense subset).