Understanding the differential in integrals

You're right that there's something subtle going on here. For example, let's take the simpler case of defining the total mass an an integral, $M = \int dm$. This has the simple intuitive meaning of saying "break the object into tiny pieces and add up the masses of each piece". And mathematically, it's called integration with respect to a measure, and studied rigorously under the name of the Lebesgue integral.

But in introductory physics, this isn't directly useful, because we want to know how to actually carry out the integration. For example, the moment of inertia can be written as $$I = \int dI = \int r^2 dm$$ which sheds a lot of light onto what moment of inertia means. But this is useless for actual calculations because we don't have rules for integrating over these variables. What we do almost all the time is to introduce a parametrization, and then integrate with respect to the parameter.

For example, we might define $I(r)$ as the moment of inertia, only counting masses with radius $r$ or less. In that case, we really want $I(R)$, where $R$ is the radius of the object, and $$I = \int dI = \int_0^R \frac{dI}{dr} \, dr.$$ Here, $dI/dr$ is the rate of change of the moment of inertia as we count larger and larger radii, so $(dI/dr)\, dr = dI$ is the contribution of the moment of inertia due to a thin slice of thickness $dr$.

That means that in your case, the right expression for $dI$ is $$dI = r^2 dm = r^2 (\rho \cdot 2 \pi r dr) = \frac{2Mr^3}{R^2} dr$$ where $\rho$ is the density of the object. In the second expression, we're implicitly thinking of $dm$ as the amount of mass in a little radius $dr$. If you're uncomfortable with the second equality, think of $m(r)$ as a function that gives the amount of mass inside a radius $r$ (like we defined $I(r)$). Then we're simply replacing $dm$ with $(dm/dr)\, dr$ by the chain rule.