Does every group whose order is a power of a prime p contain an element of order p?

There are some results that are much stronger than that. Cauchy's theorem states that every finite group whose order is divisible by some prime $p$ has a subgroup of order $p$. And from Sylow's theorem it can be deduced (although not immediately) that if the order of the group is $p^n$ then there is one subgroup of order $p^k$ for every $k=0,1,..,n$.

One more thing, a subgroup of order $p$ must be cyclic, that is, there has to be an element of order $p$ in it. That is because by Langrange's theorem the order of every element must divide the $p$ and since it is prime then the order must be $1$ or $p$. Any element different from the identity will do the trick.

Note: Lagrange's theorem alone is not enough to prove this since it only states that if the order of the group is $p^n$ then every subgroup is of the form $p^k$. That is because what Lagrange's theorem says is that the order of every subgroup must divide the order of the group. So you actually need a little bit more I think.

Late note due to nice comments: While the result doesn't follow directly from Lagrange's theorem statement. It can be derived from it as it is nicely shown to you in other answers. So you actually can avoid appealing to a stronger result such as Cauchy's theorem since you are in a finite $p$-group (what I mean by saying that it doesn't follow directly is that Lagrange's theorem makes no reference to $p$-groups, so there is math involved).

This follows immediately from Lagrange theorem, you don't need any stronger result.

If the order of the group is $p^k$ with $k \neq 0$, then by Lagrange Theorem, the order of any element divides $p^k$.

Pick some $x \in G, x \neq e$. Then the order of $x$ is $p^m$ with $1 \leq m \leq k$. Let

$$y:=x^{p^{m-1}} \,.$$

Prove that the order of $y$ is $p$.

Not the group of order $p^0=1$!

Other than that, first prove that if the order of a group element $x$ is $mn$, then the order of $x^m$ is $n$. Then you can either show directly that if $x\in G$ and $|G|$ is finite, $x^{|G|}$ is the identity, or apply Lagrange's theorem to $\langle x \rangle$.