Show that $\int_1^{\infty } \frac{(\ln x)^2}{x^2+x+1} \, dx = \frac{8 \pi ^3}{81 \sqrt{3}}$

Shocked, shocked! that there is no contour integration yet. So, without further ado...

Note that

$$f(x) = \frac{\log^2{x}}{x^2+x+1} \implies f \left ( \frac1{x} \right ) = x^2 f(x) $$

Thus, $$\int_1^{\infty} dx \frac{\log^2{x}}{x^2+x+1} = \int_0^{1} \frac{\log^2{x}}{x^2+x+1} = \frac12 \int_0^{\infty} dx \frac{\log^2{x}}{x^2+x+1} $$

Now consider

$$\oint_C dz \frac{\log^3{z}}{z^2+z+1} $$

where $C$ is a keyhole contour of outer radius $R$ and inner radius $\epsilon$. Taking the limit as $R \to \infty$ and $\epsilon \to 0$, we get that the contour integral is equal to

$$\int_0^{\infty} dx \frac{\log^3{x} - (\log{x}+i 2 \pi)^3}{x^2+x+1} $$

or

$$-i 6 \pi \int_0^{\infty} dx \frac{\log^2{x}}{x^2+x+1} + 12 \pi^2 \int_0^{\infty} dx \frac{\log{x}}{x^2+x+1} +i 8 \pi^3 \int_0^{\infty} dx \frac{1}{x^2+x+1} $$

Note that the first integral is what we seek, the second integral is zero (by the same trick we applied above), and the third integral is relatively easy to find:

$$\int_0^{\infty} \frac{dx}{x^2+x+1} = \int_0^{\infty} \frac{dx}{(x+1/2)^2+3/4} = \frac{2}{\sqrt{3}} \left [\arctan{\frac{2}{\sqrt{3}} \left ( x+\frac12 \right )} \right ]_0^{\infty} = \frac{2 \pi}{3 \sqrt{3}}$$

The contour integral is also equal to $i 2 \pi$ times the sum of the residues at the poles of the integrand, which are at $z_+ = e^{i 2 \pi/3}$ and $z_- = e^{i 4 \pi/3}$. The sum of the residues is

$$\frac{-i 8 \pi^3/27}{i \sqrt{3}} + \frac{-i 64 \pi^3/27}{-i \sqrt{3}} = \frac{56 \pi^3}{27 \sqrt{3}}$$

Then

$$-i 6 \pi \int_0^{\infty} dx \frac{\log^2{x}}{x^2+x+1} = i 2 \pi \frac{56 \pi^3}{27 \sqrt{3}} - i 8 \pi^3 \frac{2 \pi}{3 \sqrt{3}} = -i \frac{32 \pi^4}{27 \sqrt{3}}$$

Thus,

$$\int_1^{\infty} dx \frac{\log^2{x}}{x^2+x+1} = \frac12 \int_0^{\infty} dx \frac{\log^2{x}}{x^2+x+1} = \frac{8 \pi^3}{81 \sqrt{3}} $$

Here is an approach.

Observe that $$ I:=\int_1^{\infty } \frac{(\ln x)^2}{x^2+x+1} \, dx=\int_1^{\infty } \frac{(x-1)(\ln x)^2}{x^3-1} \, dx $$ and by the change of variable $x \to 1/x$ $$ I=\int_0^1 \frac{(1-x)\color{blue}{(\ln x)^2}}{1-x^3} \, dx. \tag1 $$ Since $\displaystyle \color{red}{\partial_s^2\color{red}{ (x^s)}|_{\large s=0}}=\color{blue}{(\ln x)^2}$, one may write that $$ I=\left.\color{red}{\partial_s^2}\left(\int_0^1 \frac{x^s(1-x)}{1-x^3} \, dx\right)\right|_\color{red}{{\large s=0}} \tag2 $$ Now $$ \begin{align} \int_0^1 \frac{x^s(1-x)}{1-x^3} \, dx&=\frac13\int_0^1 \frac{u^{(s-2)/3}-u^{(s-1)/3}}{1-u} \, du\qquad (u=x^3,\,x=u^{1/3})\\\\ &=\frac13\int_0^1 \frac{(1-u^{(s-1)/3})-(1-u^{(s-2)/3})}{1-u} \, du\\\\ &=\frac13\int_0^1 \frac{1-u^{(s-1)/3}}{1-u} \, du-\frac13\int_0^1 \frac{1-u^{(s-2)/3}}{1-u} \, du\\\\ &=\frac13\psi\left(\frac{s+2}{3}\right)-\frac13\psi\left(\frac{s+1}{3}\right) \tag3 \end{align} $$ where we have used the standard integral representation for the digamma function.

Then using $(2)$, we get $$ I=\frac1{27}\psi''\!\!\left(\frac{2}{3}\right)-\frac1{27}\psi''\!\!\left(\frac{1}{3}\right)=\frac{8\pi^3}{81\sqrt{3}} $$ taking into account some special values of $\psi''$.

It can be observed that $x^{2} + x + 1 = (x-a)(x-b)$ where $a = e^{2\pi i/3}$ and $b = e^{-2\pi i/3}$. Now \begin{align} I &= \int_{1}^{\infty} \frac{ (\ln(x))^{2} }{ (x-a)(x-b) } \, dx = \frac{1}{a-b} \, \int_{1}^{\infty} \left( \frac{1}{x-a} - \frac{1}{x-b} \right) \, (\ln(x))^{2} \, dx. \end{align} From Wolfram Alpha the integral \begin{align} \int \frac{ (\ln(x))^{2} }{ x - a } dx = -2 Li_{3}\left( \frac{x}{a} \right) +2 \log(x) \, Li_{2} \left( \frac{x}{a} \right) + \log^{2}(x) \log\left( 1-\frac{x}{a} \right) \end{align} for which the integral in question becomes \begin{align} I &= \left[ \frac{-2}{a-b} \left(Li_{3}\left( \frac{x}{a} \right) - Li_{3}\left(\frac{x}{b} \right) \right) + \frac{2}{a-b} \log(x) \, \left(Li_{2} \left( \frac{x}{a} \right) - Li_{2}\left( \frac{x}{b} \right) \right) + \frac{1}{a-b} \, \log^{2}(x) \log\left( \frac{a-x}{b-x} \right) \right]_{1}^{\infty} \\ &= \frac{-2}{a-b} \left[ Li_{3}\left( \frac{1}{a} \right) - Li_{3}\left(\frac{1}{b} \right) \right]. \end{align} This can then be seen as \begin{align} I &= \frac{-2i}{\sqrt{3}} \left[ Li_{3}\left( e^{2\pi i/3} \right) - Li_{3}\left( e^{- 2\pi i/3} \right) \right] \\ &= \frac{-2 i}{\sqrt{3}} \cdot \frac{4 \pi^{3} i}{81} = \frac{8 \pi^{3}}{81 \sqrt{3}}. \end{align}

\begin{align} \int_{1}^{\infty} \frac{ (\ln(x))^{2} }{ x^{2} + x + 1 } \, dx = \frac{8 \pi^{3}}{81 \sqrt{3}}. \end{align}