Prove that $\sum_{n = 1}^{p - 1} n^{p - 1} \equiv (p - 1)! + p \pmod {p^2}$ with $p$ being an odd prime

The result can be easily proved without using Bernoulli numbers. If $a$ and $b$ are integers not divisible by an odd prime $p$, then \begin{align}(ab)^{p-1}-1=&b^{p-1}(a^{p-1}-1)+(b^{p-1}-1) \\\equiv& (a^{p-1}-1)+(b^{p-1}-1)\pmod {p^2}.\end{align} Thus \begin{align*}\sum_{n=1}^{p-1}(n^{p-1}-1)\equiv& \prod_{n=1}^{p-1}n^{p-1}-1=((p-1)!+1-1)^{p-1}-1 \\\equiv &(p-1)((p-1)!+1)(-1)^{p-2}\equiv(p-1)!+1\pmod{p^2}\end{align*} and hence the desired congriuence follows.

You may use Faulhaber's formula $$ \sum_{k=1}^p k^{p-1}=\frac{p^p}p+\frac12 p^{p-1}+\sum_{k=2}^{p-1} \frac{B_k}{k!}\,(p-1)^{\underline{k-1}}\,p^{p-k}. $$ All summands except this corresponding to $k=p-1$ are divisible by $p^2$ (they are not integers, but corresponding Bernoulli numbers do not have $p$ in denominators). For $k=p-1$ you get $pB_{p-1}$. For evaluating it modulo $p^2$ we may use the formula for Bernoulli numbers via Worpitzky numbers: $$ B_{p-1}=\sum_{k=0}^{p-1} (-1)^k\frac{k!}{k+1}\left\{\matrix{p\\k+1}\right\}. $$ All summands except corresponding to $k=0$, $k=p-1$ are divisible by $p$ (the corresponding Stirling numbers of the second kind are divisible by $p$ dues to the action of the cyclic shifts group on the partitions of $\{1,2,\dots,p\}$). For $k=0$ and $k=p-1$ we get $pB_{p-1}\equiv (p-1)!+p\pmod {p^2}$ as desired.

It is a theorem of Lerch of 1905. An elementary proof can be found in this article of Sondow : https://arxiv.org/abs/1110.3113