Evaluating $\int_0^1 \frac{\log x \log \left(1-x^4 \right)}{1+x^2}dx$

I tried substitutions and the differentiation w.r.t a paramater trick like the other posters. Another partial result, or a trail of breadcrumbs to follow, is the following. We try a series expansion, $$ \frac{\log\left(1-x^4\right)}{1+x^2} = \displaystyle \sum_{k=1}^{\infty} x^{4k}\left(x^{2} -1\right)H_k, $$ where $H_k$ are the Harmonic numbers. Then \begin{align} \int_0^1 \frac{\log x \log \left(1-x^4 \right)}{1+x^2}\ \mathrm{d}x &=\displaystyle \sum_{k=1}^{\infty}\, H_k\int_0^1 x^{4k}\left(x^{2} -1\right)\log x \ \mathrm{d}x \\ &=\displaystyle \sum_{k=1}^{\infty} \, \frac{H_k}{(4k+1)^2}-\displaystyle \sum_{k=1}^{\infty} \, \frac{H_k}{(4k+3)^2}. \end{align} These sums look very similar to the ones evaluated in this post, in which they are transformed into alternating sums. Using the same techniques, or perhaps working back from the answers, we can hopefully show that $$ \displaystyle \sum_{k=1}^{\infty} \, \frac{H_k}{(4k+1)^2} = -G\left(\frac{\pi}{4}+\frac{\log 8}{2} \right) +\frac{7}{4}\zeta(3) +\frac{\pi^3}{32} - \frac{\pi^2}{16}\log 8, $$ $$ \displaystyle \sum_{k=1}^{\infty} \, \frac{H_k}{(4k+3)^2} = -G\left(\frac{\pi}{4}-\frac{\log 8}{2} \right) +\frac{7}{4}\zeta(3) -\frac{\pi^3}{32} - \frac{\pi^2}{16}\log 8, $$ Subtracting the second from the first gives us $$ \frac{\pi^3}{16}-G\log 8. $$

The following is a proof of the formula $$S= \sum_{k=1}^{\infty} \frac{H_{k}}{ (k+a)^{2}}= \left(\gamma + \psi(a) \right) \psi_{1}(a) - \frac{\psi_{2}(a)}{2} \, , \quad a >0.$$

This formula is mentioned in a comment under Bennett Gardiner's answer.

(For $a=0$, the right side of the equation should be interpreted as a limit).

$$ \begin{align} S &= \sum_{k=1}^{\infty} \frac{H_{k}}{(k+a)^{2}} \\ &= \sum_{k=1}^{\infty} \frac{1}{(k+a)^{2}} \sum_{n=1}^{k} \frac{1}{n} \\& = \sum_{n=1}^{\infty} \frac{1}{n} \sum_{k=n}^{\infty} \frac{1}{(k+a)^2} \\ &= \sum_{n=1}^{\infty} \frac{\psi_{1}(a+n)}{n} \\ &= - \sum_{n=1}^{\infty} \frac{1}{n} \int_{0}^{1} \frac{x^{a+n-1} \ln x}{1-x} \, dx \tag{1} \\ &= - \int_{0}^{1} \frac{x^{a-1} \ln x}{1-x} \sum_{n=1}^{\infty} \frac{x^{n}}{n} \, dx \\ &= \int_{0}^{1} \frac{x^{a-1} \ln x \ln(1-x)}{1-x} \, dx \\ &= \lim_{b \to 0^{+}} \frac{\partial }{\partial a \, \partial b} B(a,b) \\ &= \small \lim_{b \to 0^{+}} \frac{\Gamma(a) \Gamma(b)}{\Gamma(a+b)} \left( \psi(a) \psi(b) - \psi(a)\psi(a+b) - \psi(b) \psi(a+b) + \psi^{2}(a+b) - \psi_{1}(a+b) \right) \tag{2} \\ &= \lim_{b \to 0^{+}} \frac{\Gamma(a)}{\Gamma(a+b)} \left( \frac{1}{b} - \gamma + \mathcal{O}(b) \right)\left( \left( \gamma \psi_{1}(a) + \psi(a) \psi_{1} (a) - \frac{\psi_{2}(a)}{2} \right)b + \mathcal{O}(b^{2}) \right) \\ &= \left(\gamma + \psi(a) \right) \psi_{1}(a) - \frac{\psi_{2}(a)}{2} \end{align}$$

---

$(1)$ https://en.wikipedia.org/wiki/Trigamma_function#Calculation

$(2)$ http://mathworld.wolfram.com/BetaFunction.html (26)

This is a partial solution.

Let us put, for $0\leq t\leq 1$,

$$F(t) = \int_0^1 \frac{\log x \log(1-tx^4)}{1+x^2} dx$$

Then

$$F'(t) = -\int_0^1 \frac{x^4\log x}{(1+x^2)(1-tx^4)} dx = -\int_0^1 \frac{x^4\log x}{1+x^2} \sum_{n=0}^\infty t^nx^{4n} dx$$

$$=-\sum_{n=0}^\infty t^{n} C_{4(n+1)}$$

where $$C_m = \int_0^1 \frac{x^{m}\log x}{1+x^2} dx.$$

One has $C_0 = -G$. Multiplying both sides of the identity $$x^m = \frac{x^m}{1+x^2} + \frac{x^{m+2}}{1+x^2}$$ by $\log x$ and integrating from $0$ to $1$, one finds the recurrence formula

$$C_m + C_{m+2} = \frac{-1}{(1+m)^2}$$

and therefore

$$C_{m+4} - C_m = \frac{-1}{(3+m)^2} + \frac{1}{(1+m)^2}.$$

Therefore,

$$C_0 = -G$$ $$C_4 = -G +1 - \frac{1}{3^2}$$ $$C_8 = -G + 1 - \frac{1}{3^2} + \frac{1}{5^2} - \frac{1}{7^2}.$$

and so on. (Remark that $C_{4n} \to 0$ by definition of $G$.) Now, remark that $F(0) = 0$, so your integral is

$$F(1) = \int_0^1 F'(t) dt = -\sum_{n=0}^\infty \frac{C_{4(n+1)}}{n+1} = -\sum_{n=1}^\infty \frac{C_{4n}}{n}.$$

Now, it should be a matter of partial summation to transform the sum $-\sum_{n=1}^\infty \frac{C_{4n}}{n}$ into $\pi^3/16 -3G\log 2$ (in a manner similar to this), but I don't see it right away. I'll think about it a bit more later.