Show that $(\binom{p^2}{p} -p ) $ is divisible by $p^5$, for every prime number $p, p\ge 5$

It's sufficient to prove that $\sum\limits_{i=1}^{p-1} \frac{1}{i} \equiv 0 \pmod {p^2}$. Denote $S=\sum\limits_{i=1}^{p-1} \frac{1}{i}$. Then, note that $$ 2S=\sum\limits_{i=1}^{p-1} \left(\frac{1}{i}+\frac{1}{p-i}\right)= p\cdot \sum\limits_{i=1}^{p-1} \frac{1}{i(p-i)}. $$ Hence, it's sufficient (since $p>3$) to prove that $$ \sum\limits_{i=1}^{p-1}\frac{1}{i(p-i)}\equiv 0\pmod p. $$ However, $$ \sum\limits_{i=1}^{p-1}\frac{1}{i(p-i)}\equiv -\sum\limits_{i=1}^{p-1}\frac{1}{i^2}\pmod p. $$ Since modulo $p$ we have $\{1,2,\ldots, p-1\}=\left\{\frac{1}{1},\frac{1}{2},\ldots, \frac{1}{p-1}\right\}$ we obtain $$ \sum\limits_{i=1}^{p-1}\frac{1}{i^2}\equiv \sum\limits_{i=1}^{p-1} i^2=\frac{p(p-1)(2p-1)}{6}\equiv 0\pmod p $$ for prime $p\geq 5$. Thus, $\sum\limits_{i=1}^{p-1} \frac{1}{i^2} \equiv 0 \pmod {p}$ and so $\sum\limits_{i=1}^{p-1} \frac{1}{i} \equiv 0 \pmod {p^2}$, as desired.

Since $$ \binom{p^2}p = \frac{p^2(p^2-1)\dotsb(p^2-(p-1))}{1\cdot2\dotsb(p-1)p}, $$ we want to show that $$ \frac{(p^2-1)\dotsb(p^2-(p-1))}{1\cdot2\dotsb(p-1)} \equiv 1 \pmod{p^4}; $$ that is, $$ (p^2-1)\dotsb(p^2-(p-1)) \equiv (p-1)! \pmod{p^4}. $$ Opening the parentheses in the LHS and observing that $(-1)(-2)\dotsb(-(p-1))=(p-1)!$, it remains to show that the sum of all $p-1$ terms divisible by $p^2$ but not by $p^4$ is $0$ mod $p^4$; that is, considering the polynomial
$$ P(x) := (x-1)(x-2)\dotsb(x-(p-1)), $$ we want to show that the coefficient of the linear term of this polynomial is $0$ mod $p^2$.

To this end we make two observations. The first is nearly trivial: $$ P(p-x) = P(x). \tag{1} $$ The second observation is that the polynomial $P(x)-(x^{p-1}-1)$ has degree at most $p-2$, while the value of this polynomial at every integer point is divisible by $p$ (hint: use Wilson's theorem for $x\equiv 0\pmod p$); consequently, $$ P(x) = (x^{p-1}-1) + pQ(x), \tag{2} $$ where $Q$ is a polynomial with integer coefficients.

Substituting (1) into (2), we get $$ (p-x)^{p-1}-1+pQ(p-x) = x^{p-1}-1+pQ(x), $$ which, in view of $(p-x)^{p-1}\equiv x^{p-1}-(p-1)px^{p-2}\pmod{p^2}$, yields $$ pQ(p-x) \equiv (p-1)px^{p-2} + pQ(x) \pmod{p^2}; $$ that is, $$ Q(p-x) \equiv Q(x)-x^{p-2} \pmod p. $$ But $Q(p-x)\equiv Q(-x)\pmod p$; therefore $$ Q(-x)\equiv Q(x)-x^{p-2}\pmod p, $$ and it follows that the linear terms of $Q(-x)$ and $Q(x)$ vanish mod $p$. Now by (2), the linear term of $P$ is divisible by $p^2$. This completes the proof.