How to evaluate $ \int_0^\infty \frac{\log x}{(x^2+a^2)^2} dx $

One thing you can do when confronted with integrals of the form

$$\int_0^{\infty} dx \, f(x) \log{x} $$

is to consider a contour integral of the form

$$\oint_C dz \, f(z) \, \log^2{z} $$

where $C$ is a keyhole contour about the positive real axis, as pictured below.

To evaluate the contour integral, we parametrize about each piece of the contour. There are four such pieces: a large arc of radius $R$, a small arc of radius $\epsilon$, and lines above and below the positive real axis.

This contour allows us to derive the integral of interest by exploiting the multivalued behavior of the log at a branch point. In this case, we define the argument of the complex numbers above the positive real axis to be zero and below to be $2 \pi$. Thus, above the real axis $z=x$ while below $z=x e^{i 2 \pi}$. This difference is crucial when taking logs.

I will let the reader perform the analysis as the outer radius $R \to \infty$ and inner radius $\epsilon \to 0$; the contour integral is then equal to

$$\int_0^{\infty} dx \, f(x) \log^2{x} - \int_0^{\infty} dx \, f(x) (\log{x}+i 2 \pi)^2 = -i 4 \pi \int_0^{\infty} dx \, f(x) \log{x} + 4 \pi^2 \int_0^{\infty} dx \, f(x) $$

By the residue theorem, the contour integral is also equal to $i 2 \pi$ times the sum of the residues at the poles $z_k$ of $f$ in the complex plane outside of the origin. Thus,

$$\int_0^{\infty} dx \, f(x) \log{x} = -i \pi \int_0^{\infty} dx \, f(x) - \frac12 \sum_k \operatorname*{Res}_{z=z_k} [f(z) \log^2{z}]$$

In the OP's case,

$$f(z) = \frac1{(z^2+a^2)^2}$$

so the poles are of order two and the residues must be computed accordingly. The OP should be able to derive

$$ \operatorname*{Res}_{z=\pm i a} \frac{\log^2{z}}{(z^2+a^2)^2} = \left[\frac{d}{dz} \frac{\log^2{z}}{(z\pm i a)^2} \right ]_{z=\pm i a} $$

Note also that the poles must have their arguments between $[0,2 \pi]$ for the residue calculation to come out correctly. In this case, we may say that the poles are at $z_{\pm}=\pm i a$, but it is important to note that $z_+ = a e^{i \pi/2}$ and $z_-=a e^{i 3 \pi/2}$.

Further, it should not escape notice that the final result is in terms of an integral over the function $f$ without the log term. You should be able to see that the integral may be evaluated in exactly the same way as the original integral by introducing a log and integrating over the keyhole contour $C$. The result is

$$\int_0^{\infty} dx \, f(x) = -\sum_k \operatorname*{Res}_{z=z_k} [f(z) \log{z}]$$

At this point the OP has everything needed to carry out the computation.

I thought that it might be instructive to add to the answer posted by @RonGordon. We note that the integral of interest $I_1(a^2)$ can be written

$$I_1(a^2)=\int_0^\infty \frac{\log^2 x}{(x^2+a^2)^2}\,dx=-\frac{dI_2(a^2)}{d(a^2)}$$

where

$$I_2(a^2)=\int_0^\infty\frac{\log^2x}{x^2+a^2}\,dx$$

Now, we can evaluate the integral $J(a^2)$

$$J(a^2)=\oint_C \frac{\log^2z}{z^2+a^2}\,dz$$

where $C$ is the key-hole contour defined in the aforementioned post. There, we have

$$\begin{align} J(a^2)&=-4\pi i\,I_2(a^2)+4\pi^2\int_0^\infty \frac{1}{x^2+a^2}\,dx \\\\ &=-4\pi i\, I_2(a^2)+\frac{2\pi^3}{a}\\\\ &=2\pi i \left(\text{Res}\left(\frac{\log^2 z}{z^2+a^2},ia\right)+\left(\text{Res}\left(\frac{\log^2 z}{z^2+a^2},-ia\right)\right)\right) \end{align}$$

Finally, after calculating the residues, and simplifying, we obtain the integral $I_2(a^2)$ whereupon differentiating with respect to $a^2$ recovers the integral of interest $I_1(a^2)$. And we are done.