How to prove that $\int_{0}^{\infty}\ln^2(x)\sin(x^2)dx=\frac{1}{32}\sqrt{\frac{\pi}{2}}(2\gamma-\pi+\ln16)^2$

$$I=\int_{0}^{\infty}\ln^2(x)\sin(x^2)dx \overset{x^2=t}=\int_0^\infty \frac{1}{2\sqrt t} \ln^2 (\sqrt t) \sin t dt =\frac18 \int_0^\infty t^{-1/2}\sin t \ln^2 t \,dt$$ Note that the last integral is the Mellin transform in $s=\frac12 $ of the sine after being differentiated twice.

See for example here a proof for: $$\int_0^\infty x^{s-1}\sin x dx= \Gamma(s) \sin\left(\frac{\pi s}{2}\right)$$ $$\Rightarrow I=\frac18\frac{d^2}{ds^2}\Gamma(s) \sin\left(\frac{\pi s}{2}\right)\bigg|_{s=\frac12}$$ It's not the end of the world to differentiate that twice since the digamma function comes in our help.

From the wiki page we have: $\Gamma'(x)=\Gamma(x)\psi(x)$ $$\Rightarrow \frac{d}{ds}\Gamma(s) \sin\left(\frac{\pi s}{2}\right)=\Gamma(s)\psi(s)\sin\left(\frac{\pi s}{2}\right) +\frac{\pi}{2}\Gamma(s)\cos\left(\frac{\pi s}{2}\right)$$ $$\Rightarrow \frac{d^2}{ds^2}\Gamma(s) \sin\left(\frac{\pi s}{2}\right)=\frac{d}{ds}\Gamma(s)\left(\psi(s)\sin\left(\frac{\pi s}{2}\right)+\frac{\pi}{2}\cos\left(\frac{\pi s}{2}\right)\right)$$ $$=\Gamma(x)\psi(x)\left(\psi(s)\sin\left(\frac{\pi s}{2}\right)+\frac{\pi}{2}\cos\left(\frac{\pi s}{2}\right)\right)+\Gamma(s)\left(\psi_1(x)\sin\left(\frac{\pi s}{2}\right)+\frac{\pi}{2}\Gamma(s)\cos\left(\frac{\pi s}{2}\right)-\frac{\pi^2}{4}\sin\left(\frac{\pi s}{2}\right)\right)$$ And now setting $s=\frac12$ we get using $\Gamma\left(\frac12\right)=\sqrt{\pi}$, $\psi\left(\frac12 \right)=-\gamma -2\ln 2 $,$\ \psi_1\left(\frac12\right)=\frac{\pi^2}{2}$ the result.

We have $$ F(\alpha)=\int_{0}^{+\infty} x^\alpha \sin(x^2)\,dx = \frac{1}{2}\int_{0}^{+\infty} x^{\alpha/2-1}\sin(x)\,dx\\=\frac{1}{2\Gamma(1-\alpha/2)}\int_{0}^{+\infty} \frac{ds}{s^{\alpha/2}(s^2+1)} $$ by the properties of the Laplace transform. The last integral can be computed through the Beta and Gamma functions, producing $$ F(\alpha) = \frac{1}{2}\,\Gamma\left(\frac{1+\alpha}{2}\right)\sin\left(\frac{\pi}{4}(1+\alpha)\right) $$ for any $\alpha$ such that $\text{Re}(\alpha)\in(-3,1)$. In order to prove the claim, it is enough to apply $\lim_{\alpha\to 0}\frac{d^2}{d\alpha^2}$ to both sides of the last identity and recall the special values of $\Gamma,\psi$ and $\psi'$ at $\frac{1}{2}$.

Let us rewrite your integral as $$\int_0^\infty \ln^2(x)\sin(x^2)dx=\frac{1}{8}\int_0^\infty \frac{\ln^2(x)\sin(x)}{\sqrt{x}}dx$$ To solve this integral, you can employ the following identity, which holds for any $p\in (0,1)$: $$\int_0^\infty x^{p-1}\sin(x)dx=\Gamma(p)\sin(\pi p/2)$$ The value of your integral can be obtained from this by differentiating both sides of this equation twice with respect to $p$, moving the derivative inside of the definite integral on the LHS, and making use of the known special values of the Digamma function.

This can be done by hand, but it requires a lot of algebra and would be best left to a CAS, as suggested in the comments.