How to evaluate $I=\int_0^{\pi/2}x^2\ln(\sin x)\ln(\cos x)\ \mathrm dx$
Tools Needed $$ \begin{align} \frac1{k(j-k)^2}&=\frac1{j^2k}-\frac1{j^2(k-j)}+\frac1{j(k-j)^2}\tag{1}\\ \frac1{k(j+k)^2}&=\frac1{j^2k}-\frac1{j^2(k+j)}-\frac1{j(k+j)^2}\tag{2}\\ \log(\sin(x))&=-\log(2)-\sum_{k=1}^\infty\frac{\cos(2kx)}{k}\tag{3}\\ \log(\cos(x))&=-\log(2)-\sum_{k=1}^\infty(-1)^k\frac{\cos(2kx)}{k}\tag{4}\\ \cos(2jx)\cos(2kx)&=\frac12\Big[\cos(2(j-k)x)+\cos(2(j+k)x)\Big]\tag{5}\\ \end{align} $$ $$ \int_0^{\pi/2}x^2\cos(2kx)\,\mathrm{d}x=\left\{ \begin{array}{} (-1)^k\frac\pi{4k^2}&\text{if }k\ne0\\ \frac{\pi^3}{24}&\text{if }k=0 \end{array}\right.\tag{6}\\ $$
Tool Use $$ \begin{align} &\int_0^{\pi/2}x^2\log(\sin(x))\log(\cos(x))\,\mathrm{d}x\\[12pt] &=\int_0^{\pi/2}x^2\left(\log(2)+\sum_{k=1}^\infty\frac{\cos(2kx)}{k}\right)\left(\log(2)+\sum_{k=1}^\infty(-1)^k\frac{\cos(2kx)}{k}\right)\,\mathrm{d}x\\[12pt] &=\log(2)^2\int_0^{\pi/2}x^2\,\mathrm{d}x +\log(2)\sum_{k=1}^\infty\frac1k\int_0^{\pi/2}x^2\cos(4kx)\,\mathrm{d}x\\ &+\sum_{j=1}^\infty\sum_{k=1}^\infty\frac{(-1)^k}{2jk}\int_0^{\pi/2}x^2\Big[\cos(2(j-k)x)+\cos(2(j+k)x)\Big]\,\mathrm{d}x\\[12pt] &=\frac{\pi^3}{24}\log(2)^2+\log(2)\frac\pi{16}\zeta(3)\\ &+\frac\pi8\sum_{j=1}^\infty\sum_{k=1}^\infty\frac{(-1)^j}{jk}\left[\mathrm{iif}\left(j=k,\frac{\pi^2}{6},\frac1{(j-k)^2}\right)+\frac1{(j+k)^2}\right]\\[12pt] &=\frac{\pi^3}{24}\log(2)^2+\log(2)\frac\pi{16}\zeta(3)\\ &+\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\sum_{k=1}^{j-1}\frac1{k(j-k)^2} +\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j^2}\frac{\pi^2}{6} +\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\sum_{k=j+1}^\infty\frac1{k(j-k)^2}\\ &+\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\sum_{k=1}^\infty\frac1{k(j+k)^2}\\[12pt] &=\frac{\pi^3}{24}\log(2)^2+\log(2)\frac\pi{16}\zeta(3)\\ &+\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\left(\frac2{j^2}H_{j-1}+\frac1jH_{j-1}^{(2)}\right) -\frac{\pi^5}{576} +\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\left(-\frac1{j^2}H_j+\frac1j\frac{\pi^2}{6}\right)\\ &+\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\left(\frac1{j^2}H_j-\frac1j\frac{\pi^2}{6}+\frac1jH_j^{(2)}\right)\\[12pt] &=\frac{\pi^3}{24}\log(2)^2+\log(2)\frac\pi{16}\zeta(3)\\ &+\frac\pi8\sum_{j=1}^\infty\frac{(-1)^j}{j}\left(\frac2{j^2}H_j+\frac2jH_j^{(2)}-\frac3{j^3}\right) -\frac{\pi^5}{576}\\[12pt] &=\frac{\pi^3}{24}\log(2)^2+\log(2)\frac\pi{16}\zeta(3)+\frac{11\pi^5}{5760} +\frac\pi4\sum(-1)^j\left(\frac1{j^3}H_j+\frac1{j^2}H_j^{(2)}\right)\\[12pt] &=\frac{\pi^3}{24}\log(2)^2+\log(2)\frac\pi{16}\zeta(3)-\frac{\pi^5}{960} -\frac\pi{16}\sum_{j=1}^\infty\frac1{j^3}H_{2j}\tag{7} \end{align} $$ Numerically, $(7)$ matches the integral. I'm working on the last harmonic sum. Both numerical integration and $(7)$ yield $0.0778219793722938643380944$.
Mathematica Help
Thanks to Artes' answer on Mathematica, I have verified that these agree to 100 places.
I'm still struggling with this integral, but I guess the following result may have a chance to be helpful:
\begin{align*} \int_{0}^{\frac{\pi}{2}} x^2 \log^2 \cos x \, dx &= \frac{11 \pi^5}{1440} + \frac{\pi^3}{24} \log^2 2 + \frac{\pi}{2}\zeta(3) \log 2 \tag{1} \\ &\approx 4.2671523609840988652 \cdots. \end{align*}
To prove this, let us consider the following identity
$$ \int_{0}^{\frac{\pi}{2}} \cos^{z}x \cos wx \, dx = \frac{\pi}{2^{z+1}} \binom{z}{\frac{z+w}{2}}.$$
You can find the proof of this identity at here. Thus it follows that
$$ \int_{0}^{\frac{\pi}{2}} x^2 \log^2 \cos x \, dx = - \left. \frac{\partial^4}{\partial z^2 \partial w^2} \frac{\pi}{2^{z+1}} \binom{z}{\frac{z+w}{2}} \right|_{(z, w) = (0, 0)}. $$
Performing a bunch of calculations, we obtain $(1)$. Similar idea shows that
$$ \int_{0}^{\frac{\pi}{2}} \log^2 \cos x \, dx = \left. \frac{\partial^2}{\partial z^2} \frac{\pi}{2^{z+1}} \binom{z}{\frac{z+w}{2}} \right|_{(z, w) = (0, 0)} = \frac{\pi^3}{24} + \frac{\pi}{2}\log 2. \tag{2} $$
Indeed, starting from the identity
$$ \log^2 \left( \frac{\sin 2x}{2} \right) = \log^2 \cos x + \log^2 \sin x + 2\log \cos x \log \sin x, $$
I obtained
\begin{align*}I &= -\frac{7}{8}\int_{0}^{\frac{\pi}{2}} x^2 \log^2 \cos x \, dx + \frac{\pi}{2} \int_{0}^{\frac{\pi}{2}} x \log^2 \cos x \, dx - \frac{3\pi^2}{32} \int_{0}^{\frac{\pi}{2}} \log^2 \cos x \, dx \\ &\quad -\frac{\log 2}{8}\int_{0}^{\pi} x^2 \log \sin x \, dx + \frac{\pi^3}{48} \log^2 2 \\ &\approx 0.077821979372293864338\cdots. \end{align*}
From the identity
$$ \log \sin x = -\log 2 - \sum_{n=1}^{\infty} \frac{\cos 2nx}{n}, $$
we obtain
$$\int_{0}^{\pi} x^2 \log \sin x \, dx = -\frac{\pi}{2} \zeta (3) - \frac{\pi^3}{3} \log 2. \tag{3}$$
Putting $(1)$, $(2)$ and $(3)$ together, I was able to derive
\begin{align*}I &= -\frac{61 \pi^5}{5760} - \frac{3\pi}{8} \zeta (3) \log 2 -\frac{\pi^3}{48} \log^2 2 + \frac{\pi}{2} \int_{0}^{\frac{\pi}{2}} x \log^2 \cos x \, dx. \end{align*}
I'm not sure if this formula will be helpful, since the last remaining integral seems to defy my techniques.
Related problem: (I), (II). The contribution of this post is to evaluate the integral
$$ I = \int_0^{\pi/2}\ln(\sin x)\ln(\cos x)dx $$
symbolically. Now to find $I$, we use the first change of variables $ t = \sin(x) $ which results in
$$ I = \frac{1}{2}\int_{0}^{1}\frac{\ln(t)\ln(1-t^2)}{\sqrt{1-t^2}}dt. $$
Following it by the change of variables $u=t^2$ gives
$$ I = \frac{1}{8}\int_{0}^{1}\frac{ \ln(u) \ln(1-u) }{ \sqrt{u} \sqrt{1-u} } du .$$
To evaluate the last integral, we consider the integral
$$ F = \frac{1}{8}\int_{0}^{1}u^{a-\frac{1}{2}} (1-u)^{b-\frac{1}{2}} du = \beta(a+1/2,b+1/2) ,$$
where $\beta(u,v)$ is the beta function.
$$ \implies I = D_{b}\,D_{a} \beta(a+1/2,b+1/2)|_{a=0,b=0}= \frac{\pi}{48} \, \left( 24\, \left( \ln \left( 2 \right)\right)^{2} -{\pi }^{2} \right),$$
where $D_a=\frac{\partial }{\partial a}$ and $D_b=\frac{\partial }{\partial b}$.