Integral $I:=\int_0^1 \frac{\log^2 x}{x^2-x+1}\mathrm dx=\frac{10\pi^3}{81 \sqrt 3}$
Consider the integral \begin{align} I = \int_{0}^{1} \frac{\ln^{2}(x)}{1 - x + x^{2}} \ dx \end{align} Now consider the factorization of $x^{2} - x + 1$ which is $(x - a)(x-b)$ where $a$ and $b$ are $e^{\pi i/3}$ and $e^{-\pi i/3}$, respectively. With this in mind it is seen that \begin{align} \frac{1}{x^{2} - x + 1} = \frac{1}{a-b} \left( \frac{1}{x - a} - \frac{1}{x-b} \right). \end{align} This can also be expanded into series form and is \begin{align} \frac{1}{x^{2} - x + 1} = \frac{1}{a-b} \ \sum_{n=0}^{\infty} \left( - \frac{1}{a^{n+1}} + \frac{1}{b^{n+1}} \right) x^{n}. \end{align} Now consider the integral \begin{align} I_{n} &= \int_{0}^{1} x^{n} \ln^{2}(x) \ dx = \partial_{n}^{2} \int_{0}^{1} x^{n} \ dx \\ &= \partial_{n}^{2} \left( \frac{1}{n+1} \right) \\ &= \frac{2}{(n+1)^{3}}. \end{align}
Since the components are built the desired integral is seen as the following. \begin{align} I &= \int_{0}^{1} \frac{\ln^{2}(x)}{1 - x + x^{2}} \ dx \\ &= \frac{1}{a-b} \ \sum_{n=0}^{\infty} \left( - \frac{1}{a^{n+1}} + \frac{1}{b^{n+1}} \right) \ \int_{0}^{1} x^{n} \ln^{2}(x) \ dx \\ &= \frac{1}{a-b} \ \sum_{n=0}^{\infty} \left( - \frac{1}{a^{n+1}} + \frac{1}{b^{n+1}} \right) \frac{2}{(n+1)^{3}} \\ &= \frac{2}{a-b} \ \sum_{n=1}^{\infty} \left( - \frac{1}{a^{n}} + \frac{1}{b^{n}} \right) \frac{1}{n^{3}} \\ &= \frac{2}{a-b} \ \sum_{n=1}^{\infty} \left( \frac{a^{n}-b^{n}}{(ab)^{n}} \right) \frac{1}{n^{3}} \\ &= \frac{2}{a-b} \ \sum_{n=1}^{\infty} \left( \frac{a^{n}}{n^{3}} - \frac{b^{n}}{n^{3}} \right) \\ &= \frac{2}{a-b} \left[ Li_{3} (a) - Li_{3}(b) \right], \end{align} where $Li_{3}(x)$ is the trilogarithm. Utilizing the results \begin{align} Li_{3}(a) &= Li_{3}(e^{\pi i/3}) = \frac{1}{3} \zeta(3) + \frac{5 \pi^{3} i }{162} \\ Li_{3}(b) &= Li_{3}(e^{-\pi i/3}) = \frac{1}{3} \zeta(3) - \frac{5 \pi^{3} i }{162} \\ a-b &= e^{\pi i /3} - e^{- \pi i/3} = \sqrt{3} i \end{align} then \begin{align} I &= \frac{2}{\sqrt{3} i} \cdot \frac{5 \pi^{3} i}{81} = \frac{10 \pi^{3}}{81 \sqrt{3}}. \end{align} Hence \begin{align} \int_{0}^{1} \frac{\ln^{2}(x)}{1 - x + x^{2}} \ dx = \frac{10 \pi^{3}}{81 \sqrt{3}} . \end{align}
Substituting $u = 1/x$ and averaging with the original integral gives that $$ \int_0^{1} \frac{\ln^2 x}{x^2+2x\cos\varphi+1} dx = \frac{1}{2}\int_0^{\infty} \frac{\ln^2 x}{x^2+2x\cos\varphi+1} dx $$
For real $\varphi$ and suitable complex $a$, we can prove using contours that $$ I(a) := \int_0^{\infty} \frac{x^{a}}{x^2+2x\cos\varphi+1} dx = \frac{\pi}{\sin(\pi a)} \frac{\sin(a\varphi)}{\sin(\varphi)}$$
Differentiating twice gives (and this is the most tedious part of the calculation) $$ I''(a) := \int_0^{\infty} \frac{x^{a}\ln^2x}{x^2+2x\cos\varphi+1} dx = \frac{2\pi}{\sin \varphi} \left[ -\phi ^2 \csc (\pi a) \sin (a \phi )+\pi ^2 \csc ^3(\pi a) \sin (a \phi )-2 \pi \phi \cot (\pi a) \csc (\pi a) \cos (a \phi )+\pi ^2 \cot ^2(\pi a) \csc (\pi a) \sin (a \phi ) \right] $$ Letting $a$ tend to $0$ gives $I''(0) = \dfrac{2\varphi \left(\pi ^2-\varphi ^2\right)}{3 \sin \varphi}$. For the integral under discussion, we have $\cos \varphi = -1/2$ so we may choose $\varphi = 2\pi/3$. This gives $I''(0) =\dfrac{20 \pi ^3}{81\sqrt 3}$. Remembering the factor $1/2$ gives the desired answer.
Real Part
Substituting $x\mapsto1/x$ says $$ \int_0^1\frac{\log(x)^2}{x^2-x+1}\mathrm{d}x =\int_1^\infty\frac{\log(x)^2}{x^2-x+1}\mathrm{d}x\tag{1} $$ Therefore, $$ \int_0^1\frac{\log(x)^2}{x^2-x+1}\mathrm{d}x =\frac12\int_0^\infty\frac{\log(x)^2}{x^2-x+1}\mathrm{d}x\tag{2} $$
Contour Integration Part
Putting the branch cut for $\log(z)$ along the positive real axis, and using the contour $$ \gamma=[0,R]e^{i\epsilon}\cup Re^{i[\epsilon,2\pi-\epsilon]}\cup[R,0]e^{-i\epsilon}\tag{3} $$ as $R\to\infty$ and $\epsilon\to0$, $\log(z)=\log(x)$ on the outbound segment and $\log(z)=\log(x)+2\pi i$ on the inbound segment and he integral around huge circular arc vanishes. Therefore, $$ \begin{align} \int_\gamma\frac{\log(z)^3}{z^2-z+1}\mathrm{d}z &=\int_0^\infty\frac{-6\pi i\log(x)^2+12\pi^2\log(x)+8\pi^3i}{x^2-x+1}\mathrm{d}x\\ &=\frac{124\pi^3}{27\sqrt3}\cdot2\pi i\tag{4} \end{align} $$ where $\dfrac{124\pi^3}{27\sqrt3}$ is the sum of the residues of $\dfrac{\log(z)^3}{z^2-z+1}$ at $e^{i\pi/3}$ and $e^{i5\pi/3}$.
Combining Real and Complex Analysis
Therefore, using $(2)$ and the imaginary part of $(4)$, we get $$ \begin{align} \int_0^1\frac{\log(x)^2}{x^2-x+1}\mathrm{d}x &=\frac12\int_0^\infty\frac{\log(x)^2}{x^2-x+1}\mathrm{d}x\\ &=\frac12\left[-\frac{124\pi^3}{81\sqrt3} +\frac{4\pi^2}{3}\int_0^\infty\frac1{x^2-x+1}\mathrm{d}x\right]\\ &=\frac12\left[-\frac{124\pi^3}{81\sqrt3} +\frac{4\pi^2}{3}\frac{4\pi}{3\sqrt3}\right]\\ &=\frac{10\pi^3}{81\sqrt3}\tag{5} \end{align} $$