Does $\int_0^\infty \sin^2 (x^2)\, dx$ converge or diverge?

Like you noted, it is enough to investigate whether $\int_1^{\infty} \sin^2(x^2) \, dx$ converges. Performing the substitution $u = x^2$, we are lead to the following integral:

$$ \frac{1}{2} \int_1^{\infty} \frac{\sin^2(u)}{\sqrt{u}} \, du = \frac{1}{4} \int_1^{\infty} \frac{1 - \cos(2u)}{\sqrt{u}} \, du. $$

Now, the integral

$$ \int_1^{\infty} \frac{\cos(2u)}{\sqrt{u}} \, du $$

converges by Dirichlet's test and since $\int_1^{\infty} \frac{1}{\sqrt{u}} \, du$ diverges, the original integral diverges.

Heuristically, when $x^2$ is large, $\sin^2(x^2)>\frac12$ about half of the time, and it is never negative. So for large $t$, $\int_0^t \sin^2(x^2)\,dx$ will grow at an asymptotic rate of at least $+\frac 14$, and therefore it diverges for $t\to+\infty$.

$$I=\int_0^\infty\sin^2(x^2)\,dx = \int_0^\infty \frac{\sin^2(z)} {2\sqrt{z}} \, dz = \int_0^\infty \frac1 {2\sqrt{z}} \,dz + \underbrace{\Re\int_0^\infty \frac{e^{2 iz}}{2\sqrt{z}} \, dz}_{\text{finite}}$$

which is clearly divergent (the convergence of the second part after the last equality sign is, for example, an easy application of Cauchy's theorem )