Evaluate the integral $I=\int_{0}^{\infty}\frac{\ln^3{x}}{(1+x^2)(1+x)^2}dx$

You can use the residue theorem. Consider the integral

$$\oint_C dz \frac{\log^4{z}}{(1+z^2)(1+z)^2}$$

where $C$ is a keyhole contour about the positive real axis, so that $\arg{z} \in [0,2 \pi)$. $C$ has an outer radius of $R$, and an inner radius of $\epsilon$. The magnitude of the integral vanishes along the outer arc as $2 \pi \log^4{R}/R^3$ as $R \to \infty$ and along the inner arc as $\epsilon \log^4{\epsilon}$ as $\epsilon \to 0$. Thus the contour integral is equal to, in these limits

$$\int_0^{\infty} dx \frac{\log^4{x}-(\log{x}+i 2 \pi)^4}{(1+x^2)(1+x)^2}$$

which, when expanded, is equal to

$$-i 8 \pi \int_0^{\infty} dx \frac{\log^3{x}}{(1+x^2)(1+x)^2}+ 24 \pi^2 \int_0^{\infty} dx \frac{\log^2{x}}{(1+x^2)(1+x)^2}\\+i 32 \pi^3 \int_0^{\infty} dx \frac{\log{x}}{(1+x^2)(1+x)^2}-16 \pi^4 \int_0^{\infty} dx \frac{1}{(1+x^2)(1+x)^2}$$

The contour integral is equal to $i 2 \pi$ times the sum of the residues at the poles $z_{1,2}=\pm i$ and $z_3=-1$. Note that we would then have to evaluate the integrals with lower powers of log. We may circumvent this by expressing the above equation as a system of equations for the unknown integrals. Let

$$R_j = \sum_{k=1}^3 \operatorname*{Res}_{z=z_k} \frac{\log^j{z}}{(1+z^2)(1+z)^2}$$

$$I_j = \int_0^{\infty} dx \frac{\log^j{x}}{(1+x^2)(1+x)^2}$$

Thus, by considering similar contour integrals in the complex plane, we have the following system of equations:

$$\begin{align}-i 8 \pi I_3+24 \pi^2 I_2+i 32 \pi^3 I_1-16 \pi^4 I_0 &= i 2 \pi R_4\\ -i 6 \pi I_2+12 \pi^2 I_1+i 8 \pi^3 I_0&=i 2 \pi R_3\\-i 4 \pi I_1+4 \pi^2 I_0 &= i 2 \pi R_2\\-i 2 \pi I_0 &= i 2 \pi R_1\end{align} $$

We may now solve this upper-diagonal system for the integrals in terms of the residues; we are only interested in $I_3$. Solving for $I_3$ and reexpressing in terms of the original notation, we find that our integral is

$$\int_0^{\infty} dx \frac{\log^3{x}}{(1+x^2)(1+x)^2} = \sum_{k=1}^3 \operatorname*{Res}_{z=z_k} \left [\frac{-\frac14 \log^4{z}+i \pi \log^3{z}+\pi^2 \log^2{z}}{(1+z^2)(1+z)^2} \right ]$$

Now we must evaluate the residues. At the poles $z_1=e^{i \pi/2}$ and $z_2=e^{i 3 \pi/2}$, the computation is straightforward:

$$\operatorname*{Res}_{z=e^{i \pi/2}} \left [\frac{-\frac14 \log^4{z}+i \pi \log^3{z}+\pi^2 \log^2{z}}{(1+z^2)(1+z)^2} \right ] = \\ \frac{-\frac14 (i \pi/2)^4+i \pi (i \pi/2)^3+\pi^2 (i \pi/2)^2}{2 i (1+i)^2}=\frac{9\pi^4}{256}$$

Similarly,

$$\operatorname*{Res}_{z=e^{i 3\pi/2}} \left [\frac{-\frac14 \log^4{z}+i \pi \log^3{z}+\pi^2 \log^2{z}}{(1+z^2)(1+z)^2} \right ] = \frac{9\pi^4}{256}$$

For the pole at $z=e^{i \pi}$, we must differentiate to evaluate the residue (double pole). Thus,

$$\operatorname*{Res}_{z=e^{i \pi}} \left [\frac{-\frac14 \log^4{z}+i \pi \log^3{z}+\pi^2 \log^2{z}}{(1+z^2)(1+z)^2} \right ] = \left [\frac{d}{dz} \frac{-\frac14 \log^4{z}+i \pi \log^3{z}+\pi^2 \log^2{z}}{1+z^2} \right ]_{z=e^{i \pi}}$$

which calculation I will spare you at this point, except to say that it is straightforward, and has remarkable cancellation of the imaginary part. The result of this calculation informs us that the residue is $-\pi^4/8$.

Finally, putting this all together, we have

$$\int_0^{\infty} dx \frac{\log^3{x}}{(1+x^2)(1+x)^2} = \frac{9 \pi^4}{256} + \frac{9\pi^4}{256}-\frac{\pi^4}{8} = -\frac{7 \pi^4}{128}$$

ADDENDUM

It should be understood that the expression for the integral in terms of residues is not specific to this particular integral and applies to any integral of the form

$$\int_0^{\infty} dx \, f(x) \, \log^3{x} = \sum_{k=1}^N \operatorname*{Res}_{z=z_k} \left [\left (-\frac14 \log^4{z}+i \pi \log^3{z}+\pi^2 \log^2{z}\right ) f(z) \right ]$$

where $f$ is sufficiently well behaved that the integral exists, and the $z_k$ are the poles of $f$ in the complex plane away from the positive real axis. In fact, the general procedure works for any integer power of log, and it would be interesting to generate a polynomial-type expression in log for arbitrary powers.

Here is an elementary way. First note that $$\int_1^{\infty} \dfrac{\ln^3(x) dx}{(1+x^2)(1+x)^2} = \int_1^0 \dfrac{-\ln^3(x)}{(1+1/x^2)(1+1/x)^2} \dfrac{-dx}{x^2} = \int_0^1 \dfrac{-x^2 \ln^3(x)}{(1+x^2)(1+x)^2}dx$$ Hence, your integral is $$I = \int_0^1 \dfrac{(1-x^2) \ln^3(x)}{(1+x^2)(1+x)^2}dx = \underbrace{\int_0^1 \dfrac{\ln^3(x)}{1+x}dx}_J - \overbrace{\int_0^1 \dfrac{x\ln^3(x)}{1+x^2}dx}^K$$ We have $$J = \sum_{k=0}^{\infty}(-1)^k\int_0^1 x^k \ln^3(x)dx = \sum_{k=0}^{\infty}(-1)^{k+1} \dfrac6{(k+1)^4} = -\dfrac7{120} \pi^4 \tag{$\star$}$$ We have $$K = \sum_{k=0}^{\infty}(-1)^k\int_0^1 x^{2k+1} \ln^3(x)dx = \sum_{k=0}^{\infty}(-1)^{k+1} \dfrac3{8(k+1)^4} = -\dfrac7{1920} \pi^4 \tag{$\dagger$}$$ Hence, $$\boxed{\color{red}{I = J-K = -\dfrac7{128}\pi^4}}$$ Where we used the following facts $$\sum_{k=0}^{\infty} \dfrac{(-1)^{k+1}}{(k+1)^4} = -\dfrac78 \zeta(4)$$ $$\zeta(4) = \dfrac{\pi^4}{90}$$ to simplify $(\star)$ and $(\dagger)$.

Here is a closed form

$$I=\int_{0}^{\infty}\dfrac{\ln^3{x}}{(1+x^2)(1+x)^2}dx = -\frac{7}{128} \pi^4 \sim -5.327059668.$$

You can use this technique.