Gravitational Potential of a Sphere vs Gravitational Binding Energy of a Sphere

There is a mistake in one of your formulas, $U=\frac{3 G M^2}{5 R}$ with $R$ equal to the sphere radius is the energy required to blow every tiny shred of the sphere apart so that its pieces no longer interact gravitationally, as you said, while $V$ as given above with $r$ equal to distance from the sphere center describes how the sphere interacts with other (celestial) bodies, i.e test particles moving in the sphere's gravitational field feel $V$.

To elaborate: the gravitational field around a point mass and around an object that's spherically symmetric is the same outside of the object due to symmetry considerations, which is why $V$ agrees with the formula for the gravitational potential between 2 masses.

For a ball with uniform/constant mass-density $\rho$, one has

$$\tag{1} \frac{M}{R^3}~=~\frac{4\pi\rho}{3}~=~\frac{m}{r^3}. $$

The gravitational potential is

$$\tag{2} V(r)~=~-\frac{Gm}{r}. $$

The infinitesimal gravitational potential self-energy is

$$\tag{3} dU ~=~ V(r)dm~\stackrel{(2)}{=}~-\frac{Gm}{r} dm~\stackrel{(1)}{=}~-\frac{GM^{\frac{1}{3}}}{R} m^{\frac{2}{3}}~dm. $$

Hence the integrated gravitational potential self-energy is

$$\tag{4} U~=~\int \! dU ~\stackrel{(3)}{=}~ -\frac{GM^{\frac{1}{3}}}{R} \int_0^M m^{\frac{2}{3}}~dm~=~-\frac{3}{5}\frac{GM^2}{R}. $$