Convergence of the integral $\int _0^\infty \ln^2x\sin(x^2)\,dx$

There is no real difficulty at $0$, since near $0$ the function $\sin(x^2)$ behaves like $x^2$, so $\lim_{x\to 0^+}\ln^2 x\sin(x^2)=0$. So we examine $$\int_1^B (\ln^2 x)( \sin(x^2))\,dx.\tag{1}$$ Rewrite as $$\int_1^B \frac{\ln^2 x}{2x} 2x \sin(x^2)\,dx,$$ and use integration by parts, letting $u=\frac{\ln^2 x}{2x}$ and $dv=2x\sin(x^2)\,dx$. Then $du=\frac{2\ln x-\ln^2 x}{2x^2}\,dx$ and we can take $v=-\cos(x^2)$. Thus our integral (1) is $$\left.\left(-\frac{\ln^2 x}{2x}\cos(x^2)\right)\right|_1^B +\int_1^B \frac{2\ln x-\ln^2 x}{2x^2}\cos(x^2)\,dx.$$ The first part gives no problem, indeed it vanishes as $B\to\infty$. The remaining integral has a (finite) limit as $B\to\infty$, because $\cos(x^2)$ is bounded and the $2x^2$ in tthe denominator crushes the $\ln$ terms in the numerator.

It follows that our original integral converges.

The textbook's answer is wrong, and there is no way to prove that this integral diverges. Instead, there are ways to establish its convergence.

Since the integrand is continuous on segment $[0,1]$, it suffices to verify convergence on $(1,\infty)$, which can be established by substituting $x=\sqrt{t}$ followed by integrating by parts, thus reducing the integral to an absolutely convergent one:$$\int\limits_1^{\infty}(\ln{x})^2\sin(x^2)\,dx=\frac{1}{8}\cdot\!\!\int\limits_1^{\infty}\frac{(\ln{t})^2}{\sqrt{t}}\!\cdot \sin{t}\,dt=-\frac{1}{8}\cdot\!\!\int\limits_1^{\infty}\frac{(\ln{t})^2}{\sqrt{t}}\,d(\cos{t})=\dots$$