Finding $\int^{\infty}_{0}\frac{\ln^2(x)}{(1-x^2)^2} dx$

You're definetly on the right track with that substitution of $x=\frac1t$

Basically we have: $$I=\int^{\infty}_{0}\frac{\ln^2(x)}{(1-x^2)^2}dx=\int_0^\infty \frac{x^2\ln^2 x}{(1-x^2)^2}dx$$ Now what if we add them up? $$2I=\int_0^\infty \ln^2 x \frac{1+x^2}{(1-x^2)^2}dx$$ If you don't know how to deal easily with the integral $$\int \frac{1+x^2}{(1-x^2)^2}dx=\frac{x}{1-x^2}+C$$ I recommend you to take a look here.

Anyway we have, integrating by parts: $$2I= \underbrace{\frac{x}{1-x^2}\ln^2x \bigg|_0^\infty}_{=0} +2\underbrace{\int_0^\infty \frac{\ln x}{x^2-1}dx}_{\large =\frac{\pi^2}{4}}$$ $$\Rightarrow 2I= 2\cdot \frac{\pi^2}{4} \Rightarrow I=\frac{\pi^2}{4}$$ For the last integral see here for example.

This is a variant of TheSimpliFire's approach given in a comment.

By letting $x=e^t$ we get $$\begin{align*} \int_0^\infty\frac{\ln^2(x)}{(1-x^2)^2}\,dx &=\int_{-\infty}^\infty\frac{t^2e^{t}}{(1-e^{2t})^2}\,dt\\ &=\int_{0}^{+\infty}\frac{t^2e^{-t}}{(1-e^{-2t})^2}\,dt+\int_{0}^\infty\frac{t^2e^{-3t}}{(1-e^{-2t})^2}\,dt\\ &=\sum_{n=0}^\infty (1+n)\int_0^\infty t^2e^{-(2n+1)t}\,dt+\sum_{n=0}^\infty (1+n)\int_0^\infty t^2e^{-(2n+3)t}\,dt\\ &=\sum_{n=0}^\infty\frac{2(1+n)}{(2n+1)^3}+\sum_{n=0}^\infty\frac{2(1+n)}{(2n+3)^3}\\ &=\left(\sum_{n=0}^\infty\frac{1}{(2n+1)^2}+\sum_{n=0}^\infty\frac{1}{(2n+1)^3}\right)+\left(\sum_{n=0}^\infty\frac{1}{(2n+1)^2}-\sum_{n=0}^\infty\frac{1}{(2n+1)^3}\right)\\ &=2\sum_{n=0}^\infty\frac{1}{(2n+1)^2}=2\left(\sum_{n=0}^\infty\frac{1}{n^2}-\sum_{n=0}^\infty\frac{1}{(2n)^2}\right)\\&=2\left(1-\frac{1}{4}\right)\sum_{n=0}^\infty\frac{1}{n^2}=\frac{3}{2}\cdot \frac{\pi^2}{6}=\frac{\pi^2}{4}. \end{align*}$$