Using the substitution method for a simple integral
$\int \frac{1}{2x}u du$ = $\int \frac{1}{2\sqrt{u}}\cdot u du$ = $ \int \frac{\sqrt{u}}{2}du$ = $u^{\frac{3}{2}}\cdot\frac{2}{3}\cdot \frac{1}{2} + C$
$= \frac{u^{\frac 3 2}}{3} + C = \frac{x^3}{3} + C$ as required.
You have one mistake. If you are changing $dx$ into $du$, then you need to convert all terms containing $x$ into $u$.
So we have,
$$\int x^2 dx = \int u \frac{1}{2\sqrt u} du$$
$$= \frac 12 \int \frac{1}{\sqrt{u}}\cdot u du = \frac 12 \int \sqrt u du$$
$$= \frac{1}{2} \cdot u^{\frac{3}{2}}\cdot\frac{2}{3}+ C$$
$$= \frac{u^{\frac 3 2}}{3} + C = \frac{x^3}{3} + C$$
if you set $$u=x^2$$ then we have $$x=\pm \sqrt{u}$$ and $$dx=\pm\frac{1}{2\sqrt{u}}du$$