How to prove that $n\log n = O(n^2)$?

From your notation, it looks like we are assuming that $n\in\mathbb{N}$. In fact, I'll be more general and just say $n\in(0,\infty)$.

Then $\log n<n$, since $n < 1+n < e^n$ by its Taylor series.

Thus, $n\log n<n^2$ for all $n\in(0,\infty)$.

As a consequence, $n\log n = O(n^2)$.

Prove that $n\log n =O(n^2)$ is equivalent to prove that the limit $$\lim_{n\to\infty}\frac{n\log n}{n^2}=\lim_{n\to\infty}\frac{\log n}{n}$$ is finite which is the case by simple applying the l'Hôpital theorem to find that the desired limit is $0$.

Remark Since this limit is $0$ we have precisely $$n\log n= o(n^2)\quad \text{little o}$$