prove $\int_0^\infty \frac{\log^2(x)}{x^2+1}\mathrm dx=\frac{\pi^3}{8}$ with real methods

You can also write it as: $$I=\int_0^1\frac{\log^2(x)}{x^2+1}dx+\int_1^\infty \frac{\log^2(x)}{x^2+1}dx$$ In the second one do a $x\rightarrow \frac{1}{x}$ $$\Rightarrow I=2\int_0^1 \frac{\ln^2 x}{1+x^2}dx =2\sum_{n=0}^\infty (-1)^n\int_0^1 x^{2n}\ln^2 xdx$$$$\int_0^1 t^a dt=\frac{1}{a+1}\Rightarrow \int_0^1 t^a \ln^k tdt=\frac{d^k}{da^k} \left(\frac{1}{a+1}\right)=\frac{(-1)^k k!}{(a+1)^{k+1}}$$ $$\Rightarrow I=4\sum_{n=0}^\infty \frac{(-1)^{n} }{(2n+1)^3}=4\cdot\frac{\pi^3}{32}=\frac{\pi^3}{8}$$

\begin{align} J&=\int_0^\infty\frac{\ln^2 x}{x^2+1}\mathrm dx\\ K&=\int_0^\infty\frac{\ln(x)}{x^2+1}\mathrm dx\\ K^2&=\int_0^\infty\frac{\ln x}{x^2+1}\mathrm dx\int_0^\infty\frac{\ln y}{y^2+1}\mathrm dy\\ &=\int_0^\infty \int_0^\infty\frac{\ln x\ln y}{(x^2+1)(y^2+1)}\mathrm dx\mathrm dy\\ &=\frac{1}{2}\int_0^\infty \int_0^\infty\frac{\ln(xy)^2-\ln^2 x-\ln^2 y}{(x^2+1)(y^2+1)}\mathrm dx\mathrm dy\\ &=\frac{1}{2}\int_0^\infty \int_0^\infty\frac{\ln(xy)^2}{(x^2+1)(y^2+1)}\mathrm dx\mathrm dy-\int_0^\infty \int_0^\infty\frac{\ln^2 x}{(x^2+1)(y^2+1)}\mathrm dx\mathrm dy\\ &=\frac{1}{2}\int_0^\infty \frac{1}{y^2+1}\left(\int_0^\infty\frac{\ln(xy)^2}{x^2+1}\mathrm dx\right)\mathrm dy-\frac{\pi}{2}J \end{align}

Perform the change of variable $u=xy$ ($x$ is the variable)

\begin{align}K^2&=\frac{1}{2}\int_0^\infty \int_0^\infty\frac{y\ln^2 u}{(y^2+1)(y^2+u^2)}\mathrm du\mathrm dy-\frac{\pi}{2}J\\ &=\frac{1}{4}\int_0^\infty\frac{\ln^2 u}{1-u^2}\left[\ln\left(\frac{u^2+y^2}{1+y^2}\right)\right]_{y=0}^{y=\infty}\mathrm du-\frac{\pi}{2}J\\ &=-\frac{1}{2}\int_0^\infty\frac{\ln^3 u}{1-u^2}\mathrm du\\ &=-\frac{1}{2}\left(\int_0^1\frac{\ln^3 u}{1-u^2}\mathrm du+\int_1^\infty\frac{\ln^3 u}{1-u^2}\mathrm du\right)-\frac{\pi}{2}J\\ \end{align}

In the second integral perform the change of variable $v=\dfrac{1}{u}$

\begin{align}K^2&=-\int_0^1\frac{\ln^3 u}{1-u^2}\mathrm du-\frac{\pi}{2}J\\ &=\frac{1}{2}\int_0^1\frac{2u\ln^3 u}{1-u^2}\mathrm du-\int_0^1\frac{\ln^3 u}{1-u}\mathrm du-\frac{\pi}{2}J \\\end{align}

In the first integral perform the change of variable $v=u^2$,

\begin{align}K^2&=\frac{1}{16}\int_0^1\frac{\ln^3 u}{1-u}\mathrm du-\int_0^1\frac{\ln^3 u}{1-u}\mathrm du-\frac{\pi}{2}J\\ &=-\frac{15}{16}\int_0^1\frac{\ln^3 u}{1-u}\mathrm du-\frac{\pi}{2}J\\ &=-\frac{15}{16}\times -6\zeta(4)-\frac{\pi}{2}J\\ &=\frac{45}{8}\zeta(4)-\frac{\pi}{2}J\\ \end{align}

Moreover, $K=0$ (perform the change of variable $v=\frac{1}{x}$)

Therefore,

\begin{align}J&=\frac{45}{4\pi}\zeta(4)\\ &=\frac{45}{4\pi}\times \frac{\pi^4}{90}\\ &=\boxed{\frac{\pi^3}{8}} \end{align}

Recall the formula $$\int_0^\infty \frac{t^{x}}{1+t^2}dt=\frac{\pi}{2}\sec\frac{\pi x}{2}$$ Differentiating this formula twice with respect to $x$ yields the answer.

To derive this formula, use the following sequence of substitutions, the Beta Function, and the reflection formula of the Gamma Function: $$\begin{align} \int_0^\infty \frac{t^x}{1+t^2}dt &=\frac{1}{2}\int_0^\infty \frac{t^{(x-1)/2}}{1+t}dt \\ &=\frac{1}{2}\int_0^1 t^{(x-1)/2}(1-t)^{(-1-x)/2}dt\tag{1} \\ &=\frac{1}{2}\frac{\Gamma(\frac{1+x}{2})\Gamma(\frac{1-x}{2})}{\Gamma(1)} \\ &=\frac{\pi}{2}\sec\frac{\pi x}{2} \end{align}$$ where $(1)$was obtained by substituting $t\mapsto \frac{t}{1-t}$.