Can Potential Energy be found by Energy Density?

After reading the comment you have made below, I came to realize that the real deal here is that the electric field is not varying as $x$ outside the sphere.

The problem here is not the integral (I apologize for that as I did not cross-check). In fact, it seems you are not very clear with the concept of energy density.

Energy density is the energy per unit volume. What is important for this question is to realise is that you can use the energy density as a function of $x$ as long as it agrees with the value the electric field would have inside the sphere, and then outside the sphere.

Clearly, as you have derived the E-field is varying directly as $x$ inside. If you apply Gauss' law for a region outside the sphere, you get that the electric field varies as $1/x^2$.

These fields are clearly different in their functional forms. So what you need to do while calculating the potential energy of the sphere is to add the contribution inside and outside the sphere.

i.e. $ U_1 = \int \Omega_1 dV$ having the limits of $x=0$ to $x=R$ (inside the sphere)

And $U_2 = \int \Omega_2 dV$ having the limits of $x=R$ to $x \to \infty$.

The required potential energy will be: $U=U_1 +U_2$. You would find that $U_1$ is having a number factor of 1/40 as you have found already. And $U_2$ would have a factor of 1/8. Add them together and you get $\frac{3Q^2}{20 \pi \epsilon R}$

This can be proven.

Potential energy = $\frac{1}{2}\int VdQ$

$dQ=\rho d\tau$ where $d\tau$ is the volume element.

So $PE=\frac{1}{2}\int V\rho d\tau=\frac{1}{2}\int \epsilon_0V(\nabla \cdot \vec{E}) d\tau=\frac{-\epsilon_0}{2}\int V\nabla^2Vd\tau$

This because by Gauss' Law, $\nabla \cdot \vec{E}=\rho/\epsilon_0$

That $\nabla \times\vec{E}=-\frac{\partial \vec{B}}{\partial t}=0$ in Electro statics implies $\vec{E}=-\nabla V$ for some scalar V.

Now we can invoke a vector identity:

$$V\nabla^2V=\nabla \cdot (V\nabla V)-(\nabla V \cdot \nabla V)$$

So : $$PE=\frac{-\epsilon_0}{2}\int \nabla \cdot (V\nabla V) d\tau+\frac{\epsilon_0}{2}\int ( \nabla V \cdot \nabla V) d\tau$$

Now by Gauss's Law the first volume integral can be expressed as an area integral over a sphere of infinite radius, but gradient of V is zero infinitely far away. Therefor, the second integral contributes all of the PE.

So $PE=\int \frac{\epsilon_0}{2}E^2 d\tau$

As to why your integrals came out different, they are very tricky.

You need to remember the factor of 1/2 when integrating for potential energy.

Your limits of integration are different as well. Whereas your limits of integration are 0 to R, R being the radius of the sphere for the potential energy, your energy density integral is over all space. Further, the electric field is proportial to $r$, distance from center of the sphere, within the sphere and $1/r^2$ when integrating the density outside of the sphere.

Sign errors can also be a problem.