Big O estimate of $\frac{(5x^5+3x^2)(x\log x+x)}{10x^3}$.

$O(x^3 \log x)$ will do it.

The function is not bounded by a constant multiple of $x^3$ because $\log x \to\infty$ as $x\to\infty.$

Not quite, you see that $x^3\log x$ is bigger than $x^3$. So the function $$\frac{(5x^5+3x^2)(x\log x + x)}{10x^3}$$ is actually $O(x^3\log x)$.