contour integration of $\int_0^\infty \frac{\ln(x)}{x^2-1}dx$

If I'm not mistaken, you've done your substitutions incorrectly. For the straight-line segment of the integral in the upper half-plane (call it $\gamma_1$), for example, the integral is $$ I_1 = \int_{\gamma_1} \frac{\ln(z) dz}{z^2 - 1} $$ To turn this into an integral over a single real parameter $x$, we let $z = x e^{i \pi/3}$, and integrate from $x = r$ to $R$: $$ I_1 = \int_r^R \frac{\ln (x e^{i \pi/3}) (dx e^{i \pi/3})}{x^2 e^{2i \pi/3} - 1} = e^{i \pi/3} \int_r^R \frac{\ln |x| + i \pi/3}{x^2 e^{2i \pi/3} - 1} \, dx. $$ But the factor of $e^{2i \pi/3}$ multiplying $x^2$ in the denominator means that $I_1$ is not related in a simple way to the integral $I$ you're trying to calculate. So I believe this method is a dead end.

\begin{eqnarray*} \int_0^\infty\int_0^\infty\frac{dxdy}{(1+y)(1+x^2y)} &=&\int_0^\infty\frac{dy}{1+y}\int_0^\infty\frac{dx}{1+x^2y}\\ &=&\int_0^\infty\frac{dy}{1+y}\int_0^\infty\frac{1}{y}\frac{dx}{\frac{1}{y}+x^2}\\ &=&\int_0^\infty\frac{dy}{1+y}\int_0^\infty\frac{1}{y}\frac{1}{\frac{1}{\sqrt{y}}}arctan \frac{x}{\frac{1}{\sqrt{y}}}|_0^\infty\\ &=&\frac{\pi}{2}\int_0^\infty\frac{dy}{\sqrt{y}(1+y)}\\ &=&\frac{\pi}{2}2\int_0^\infty\frac{dt}{1+t^2}\\ &=&\frac{\pi^2}{2} \end{eqnarray*} \begin{eqnarray*} \int_0^\infty\int_0^\infty\frac{dxdy}{(1+y)(1+x^2y)} &=&\int_0^\infty\frac{dx}{1-x^2}\int_0^\infty\left(\frac{x^2}{1+x^2y}-\frac{1}{1+y}\right)dy\\ &=&\int_0^\infty\frac{1}{1-x^2}\left(\ln\frac{1+x^2y}{1+y}\big|_0^\infty\right)dx\\ &=&2\int_0^\infty\frac{\ln x dx}{1-x^2} \end{eqnarray*} Thus \begin{eqnarray*} \int_0^\infty\frac{\ln x dx}{1-x^2}=\frac{\pi^2}{4}. \end{eqnarray*}