Show that $\int_0^\infty \sin\left(x^2\right)dx$ converges, but that $\int_0^\infty \sqrt{\sin^2\left(x^2\right)}dx$ does not.

Your second integral is $$ \int_0^\infty|\sin x^2|\,dx. $$ Note that $\sin t\geq1/2$ if $t\in[\frac\pi6+2k\pi,\frac{5\pi}6+2k\pi]$. So $\sin x^2\geq1/2$ if $x\in[\sqrt{\frac\pi6+2k\pi},\sqrt{\frac{5\pi}6+2k\pi}]$, $k\in\mathbb N$.

So $$ \begin{eqnarray} \int_0^\infty|\sin x^2|\,dx&\geq&\sum_{k=0}^\infty\int_{\sqrt{\frac\pi6+2k\pi}}^{\sqrt{\frac{5\pi}6+2k\pi}}|\sin x^2|\,dx\\ &\geq&\frac12\,\sum_{k=0}^\infty\left(\sqrt{\frac{5\pi}6+2k\pi}-\sqrt{\frac{\pi}6+2k\pi}\right)\\ &\geq&\frac12\,\sum_{k=0}^\infty\frac{\frac{4\pi}6}{\sqrt{\frac{5\pi}6+2k\pi}+\sqrt{\frac{\pi}6+2k\pi}}\\ &\geq&\frac12\,\sum_{k=0}^\infty\frac{\frac{4\pi}6}{2\sqrt{\frac{5\pi}6+2k\pi}}\\ &=&\frac\pi8\,\sum_{k=0}^\infty\frac{1}{\sqrt{\frac{5\pi}6+2k\pi}}=\infty \end{eqnarray} $$ (the last series clearly diverges as its terms grow as $k^{-1/2}$).

Here's a proof of the first part, using the general philosophy that if an integrand oscillates, you can often get a better sense of the size of the integral using integration by parts. Ignoring the perfectly tame integral from 0 to 1, $$ \int_1^\infty \sin(x^2)\,dx = \int_1^\infty \frac1{-2x}(-2x\sin(x^2))\,dx = \frac{\cos(x^2)}{-2x}\bigg|_1^\infty - \int_1^\infty \frac{\cos(x^2)}{2x^2}\,dx. $$ The boundary terms are finite, and the remaining integral converges absolutely by comparison to $\int x^{-2}dx$.

Note that, by change of variables we have $$\int_0^\infty \sqrt{\sin^2\left(x^2\right)}dx = \int_0^\infty |\sin\left(x^2\right)|dx =\int_0^\infty \frac{|\sin\left(x\right)|}{2 x^{1/2}}dx $$

More generally we study the following integrals

$$\varphi_1(\alpha) =\int_0^\infty \frac{\sin t}{t^\alpha}\,dt\tag{I}$$

Lemma$ \frac{\sin t}{t^\alpha} $ converges if and only if $0<\alpha<2$ and converges absolutely if and only if $1<\alpha <2$.

Therefore, $\int_0^\infty \sqrt{\sin^2\left(x^2\right)}dx = \int_0^\infty |\sin\left(x^2\right)|dx =\int_0^\infty \frac{|\sin\left(x\right)|}{2 x^{1/2}}dx $ diverges

Proof of the Lemma

case $\alpha\gt 0$

Near $t=0$, $\sin t\approx t.$ Which yields, $\frac{\sin t}{t^{\alpha}}\approx \frac{1}{t^{\alpha -1}}$ and the convergence of the integral in (I) holds nearby $t=0$ if and only if $\alpha<2 $.

Now let take into play the case where $t $ is large.

case $\alpha\leq 0$

Employing integration by part, \begin{eqnarray*} \Big| \int_{\frac{\pi}{2}}^\infty \frac{\sin t}{t^\alpha}\,dt\Big| &= & \Big| -\alpha \int_{\frac{\pi}{2}}^\infty \frac{\cos t}{t^{\alpha+1}}\,dt\Big|\\ % &\leq & \alpha \int_{\frac{\pi}{2}}^\infty \frac{ 1 }{t^{\alpha+1}}\,dt< \infty \qquad\text{since} \qquad \alpha +1>1~~\text{with} ~~\alpha >0. \end{eqnarray*} Thus for $\alpha>0 $, $\varphi_1(\alpha)$ exists if and only if $0<\alpha<2$.

We will later these are the only values of $\alpha$ which guarantee the existence of $\varphi_1$. For now let have a look on the integrability of functions under (I). In other to see that, one can quickly check the following

$$ \mathbb{R}_+ = \bigcup_{n\in\mathbb{N}} [n\pi, (n+1)\pi).$$

Then, $$\int_0^\infty \frac{|\sin t|}{t^\alpha}\,dt = \int_{0}^{\pi} \frac{\sin t}{{t}^\alpha} \,dt+ \sum_{n=1}^{\infty} \int_{n\pi}^{(n+1)\pi} \frac{|\sin t|}{t^\alpha}\,dt \\:= \int_{0}^{\pi} \frac{\sin t}{{t}^\alpha} \,dt+\sum_{n=1}^{\infty} a_n$$

With suitable change of variable ($u = t-n\pi$) we get

\begin{eqnarray*} a_n &=& \int_{0}^{\pi} \frac{\sin t}{{(t+n\pi)}^\alpha} \,dt\qquad\text{since } \sin(t+n\pi)= (-1)^n\sin t \end{eqnarray*} On the oder hand, it is also easy to check

\begin{eqnarray} \frac{2}{(n+1\pi)^\alpha} \leq a_n \leq \frac{2}{(n\pi)^\alpha}. % \end{eqnarray} These inequality together with the Riemann sums show that the series of general terms $(a_n)_n$ and $(b_n)_n$ converge if and only if $\alpha>1.$ Moreover we have seen from the foregoing that

$$\int_{0}^{\pi} \frac{\sin t}{{t}^\alpha} \,dt$$ converges only for $\alpha <2$

Taking profite of the tricks above, we get the result for the case $\alpha \leq 0$ as follows

$$\int_0^\infty \frac{\sin t}{t^\alpha}\,dt = \int_{0}^{\pi} \frac{\sin t}{{t}^\alpha} \,dt+ \sum_{n=1}^{\infty} \int_{n\pi}^{(n+1)\pi} \frac{\sin t}{t^\alpha}\,dt \\:= \int_{0}^{\pi} \frac{\sin t}{{t}^\alpha} \,dt+\sum_{n=1}^{\infty} a'_n $$

With

\begin{eqnarray*} |a'_n| &=&\left|\int_{n\pi}^{(n+1)\pi} \frac{\sin t}{{(t+n\pi)}^\alpha} \,dt\right|= \left|\int_{0}^{\pi} \frac{\sin t}{{(t+n\pi)}^\alpha} \,dt\right| \geq \frac{2}{(\pi+n\pi)^\alpha} \qquad\qquad\text{since } \sin(t+n\pi) = (-1)^n\sin t . \end{eqnarray*}
and the equalities hold in both cases when $\alpha = 0.$ Therefore, $$\lim |a'_n|= \begin{cases} 2 &~~if ~~\alpha = 0 \nonumber\\ \infty & ~~if ~~\alpha <0. \nonumber \end{cases}$$ What prove that the divergence of the series $\sum\limits_{n=0}^{\infty} a'_n$ since $a_n'\not\to 0$. Consequently the left hand side of the previous relations always diverge since $\int_{0}^{\pi} \frac{\sin t}{{t}^\alpha} \,dt $ converges for $\alpha\leq 0.$

Conclusion$ \frac{\sin t}{t^\alpha} $ converges for $0<\alpha<2$ and converges absolutely for $1<\alpha <2$.