How to prove that $\int_{1}^{\sqrt{2}+1}\frac{\ln{x}}{x^{2}-1}dx=\frac{\pi^{2}}{16}-\frac{\ln^{2}\left(\sqrt{2}+1\right)}{4}$

Put \begin{equation*} I=\int_{1}^{\sqrt{2}+1}\dfrac{\ln x}{x^2-1}\, \mathrm{d}x =[x=1/y] = \int_{\sqrt{2}-1}^{1}\dfrac{\ln y}{y^2-1}\, \mathrm{d}y.\tag{1} \end{equation*} After the substitution $ y=\frac{1-z}{1+z}$ and integration by parts we have \begin{gather*} I = \int_{0}^{\sqrt{2}-1}\dfrac{\ln\left(\frac{1-z}{1+z}\right)}{-2z}\, \mathrm{d}z = \left[-\dfrac{1}{2}\ln(z)\ln\left(\frac{1-z}{1+z}\right)\right]_{0}^{\sqrt{2}-1}+\int_{0}^{\sqrt{2}-1}\dfrac{\ln z}{z^2-1}\, \mathrm{d}z =\\[2ex]-\dfrac{1}{2}\ln^2(\sqrt{2}+1) + \int_{0}^{\sqrt{2}-1}\dfrac{\ln z}{z^2-1}\, \mathrm{d}z.\tag{2} \end{gather*} If we add (1) and (2) we get \begin{equation*} 2I = -\dfrac{1}{2}\ln^2(\sqrt{2}+1) + \int_{0}^{1}\dfrac{\ln z}{z^2-1}\, \mathrm{d}z. \end{equation*} Consequently \begin{gather*} I = -\dfrac{1}{4}\ln^2(\sqrt{2}+1) -\dfrac{1}{2}\int_{0}^{1}\left(\sum_{k=0}^{\infty}\ln(z)z^{2k}\right)\, \mathrm{d}z =\\[2ex] -\dfrac{1}{4}\ln^2(\sqrt{2}+1) -\dfrac{1}{2}\sum_{k=0}^{\infty}\int_{0}^{1}\ln(z)z^{2k}\, \mathrm{d}z =\\[2ex] -\dfrac{1}{4}\ln^2(\sqrt{2}+1) +\dfrac{1}{2}\sum_{k=0}^{\infty}\dfrac{1}{(2k+1)^2} = \dfrac{\pi^2}{16} -\dfrac{1}{4}\ln^2(\sqrt{2}+1). \end{gather*}

A Complete Solution Now

Consider $$F(s)=\int_1^s\frac{\log x}{x^2-1}\mathrm dx$$ Like before, $$F(s)=\frac12\int_1^{s}\frac{\log x}{x-1}\mathrm dx-\frac12\int_1^{s}\frac{\log x}{x+1}\mathrm dx$$ $$F(s)=-\frac12\mathrm{Li}_2(1-s)-\frac12J(s)$$ For $J(s)$, we integrate by parts with $\mathrm dv=\frac{\mathrm dx}{1+x}$ to get $$J(s)=\log(s)\log(s+1)-\int_1^s\frac{\log(1+x)}{x}\mathrm dx$$ $$J(s)=\log(s)\log(s+1)+\int_2^{1+s}\frac{\log x}{1-x}\mathrm dx$$ $$J(s)=\log(s)\log(s+1)+\mathrm{Li}_2(1-x)\bigg|_2^{1+s}$$ $$J(s)=\log(s)\log(s+1)+\mathrm{Li}_2(-s)-\mathrm{Li}_2(-1)$$ Using $\mathrm{Li}_k(-1)=(2^{1-k}-1)\zeta(k)$, $$J(s)=\log(s)\log(s+1)+\mathrm{Li}_2(-s)+\frac{\pi^2}{12}$$ Plugging in $s=1+\sqrt2$, $$J(1+\sqrt2)=\log(1+\sqrt2)\log(2+\sqrt2)+\mathrm{Li}_2(-1-\sqrt2)+\frac{\pi^2}{12}$$ Which, as The OP noted, becomes $$J(1+\sqrt2)=\log^2(1+\sqrt2)+\frac12\log(2)\log(1+\sqrt2)+\mathrm{Li}_2(-1-\sqrt2)+\frac{\pi^2}{12}$$ And again as the OP noted, $$F(1+\sqrt2)=-\frac{\pi^2}{48}-\frac14\log^2(1+\sqrt2)+\frac14\log2\log(1+\sqrt2)+\frac1{16}\log^22+\frac12\mathrm{Li}_2\bigg(\frac1{\sqrt2}\bigg)-\frac12\mathrm{Li}_2(1-\sqrt2)$$ And since $$\frac12\mathrm{Li}_2\bigg(\frac1{\sqrt2}\bigg)-\frac12\mathrm{Li}_2(1-\sqrt2)=\frac{\pi^2}{12}-\frac14\log2\log(1+\sqrt2)-\frac1{16}\log^22$$ We have $$F(1+\sqrt2)=\frac{\pi^2}{16}-\frac14\log^2(1+\sqrt2)$$

Here is one possible approach.

Enforcing a substitution of $x \mapsto 1/x$ to begin with gives \begin{align} \int_1^{1 + \sqrt{2}} \frac{\ln x}{x^2 - 1} \, dx &= -\int_{\sqrt{2}-1}^1 \frac{\ln x}{1 - x^2} \, dx = -\sum_{n = 0}^\infty \int_{\sqrt{2} - 1}^1 x^{2n} \ln x \, dx, \end{align} where we have taken advantage of the geometric sum for $1/(1 - x^2)$. Integrating by parts then leads to \begin{align} \int_1^{1 + \sqrt{2}} \frac{\ln x}{x^2 - 1} \, dx &= - \ln (1 + \sqrt{2}) \sum_{n = 0}^\infty \frac{(\sqrt{2} - 1)^{2n + 1}}{2n+1} + \sum_{n = 0}^\infty \frac{1}{(2n + 1)^2}\\ &\qquad - \sum_{n = 0}^\infty \frac{(\sqrt{2} - 1)^{2n + 1}}{(2n + 1)^2}. \tag1 \end{align}

Observing that $$\tanh^{-1} z = \sum_{n = 0}^\infty \frac{z^{2n + 1}}{2n + 1}, \qquad |z| < 1, \qquad (*)$$ the first of the sums appearing in (1) can be expressed as $$\sum_{n = 0}^\infty \frac{(\sqrt{2} - 1)^{2n + 1}}{2n+1} = \tanh^{-1} (\sqrt{2} - 1) = \frac{1}{2} \ln (1 + \sqrt{2}).$$

To find the second and third sums in (1), divide ($*$) by $x$ before integrating up from $0$ to $x$. Thus \begin{align} \sum_{n = 0}^\infty \frac{1}{2n + 1} \int_0^x t^{2n} \, dt &= \int_0^x \frac{\tanh^{-1} t}{t} \, dt\\ \sum_{n = 0}^\infty \frac{x^{2n + 1}}{(2n + 1)^2} &= \frac{1}{2} \int_0^x \ln \left (\frac{1 + t}{1 - t} \right ) \frac{dt}{t}\\ &= \frac{1}{2} \int_0^{-x} \frac{\ln (1 - t)}{t} \, dt - \frac{1}{2} \int_0^x \frac{\ln (1 - t)}{t} \, dt \end{align} or $$\sum_{n = 0}^\infty \frac{x^{2n + 1}}{(2n + 1)^2} = \frac{1}{2} \left [\operatorname{Li}_2 (x) - \operatorname{Li}_2 (-x) \right ], \qquad (**)$$ where $\operatorname{Li}_2 (x)$ is the dilogarithm function.

The second and third sums appearing in (1) can now be found using ($**$). For the second sum, setting $x = 1$ gives $$\sum_{n = 1}^\infty \frac{1}{(2n + 1)^2} = \frac{1}{2} \operatorname{Li}_2 (1) - \frac{1}{2} \operatorname{Li}_2 (-1) = \frac{1}{2} \left (\frac{\pi^2}{6} + \frac{\pi^2}{12} \right ) = \frac{\pi^2}{8},$$ where the well-known special values for the dilogarithm function at 1 and $-1$ have been used.

For the third sum, setting $x = \sqrt{2} - 1$ gives $$\sum_{n = 0}^\infty \frac{(\sqrt{2} - 1)^{2n + 1}}{(2n + 1)^2} = \frac{1}{2} \left [\operatorname{Li}_2 (\sqrt{2} - 1) - \operatorname{Li}_2 (1 - \sqrt{2}) \right ]. \qquad (\dagger)$$ It now remains to express the difference between the two dilogarithms is terms of elementary constants.

To do this, we will make use of the following identities for the dilogarithm: \begin{align} \operatorname{Li}_2 (1-x) + \operatorname{Li}_2 \left (1 - \frac{1}{x} \right ) &= -\frac{1}{2} \ln^2 x \tag2\\ \operatorname{Li}_2 (x) + \operatorname{Li}_2 (1-x) &= \frac{\pi^2}{6} - \ln x \ln (1 - x) \tag3\\ \operatorname{Li}_2 (x) + \operatorname{Li}_2 (-x) &= \frac{1}{2} \operatorname{Li}_2 (x^2) \tag4 \end{align} So here we go. \begin{align} \operatorname{Li}_2 (\sqrt{2} - 1) &= \operatorname{Li}_2 [1 - (2 - \sqrt{2})]\\ &= -\operatorname{Li}_2 \left (1 - \frac{1}{2 - \sqrt{2}} \right ) - \frac{1}{2} \ln^2 (2 - \sqrt{2}) \qquad \text{(by (2))}\\ &= - \operatorname{Li}_2 \left (-\frac{1}{\sqrt{2}} \right ) - \frac{1}{2} \ln^2 \left (\frac{\sqrt{2}}{1 + \sqrt{2}} \right )\\ &= - \operatorname{Li}_2 \left (-\frac{1}{\sqrt{2}} \right ) -\frac{1}{2} \left [\frac{1}{2} \ln 2 - \ln (1 + \sqrt{2}) \right ]^2\\ &= -\operatorname{Li}_2 \left (-\frac{1}{\sqrt{2}} \right ) - \frac{1}{8} \ln^2 2 + \frac{1}{2} \ln 2 \ln (1 + \sqrt{2}) - \frac{1}{2} \ln^2 (1 + \sqrt{2}) \end{align} And \begin{align} \operatorname{Li}_2 (1 - \sqrt{2}) &= -\operatorname{Li}_2 \left (1 - \frac{1}{\sqrt{2}} \right ) - \frac{1}{2} \ln^2 \sqrt{2} \qquad \text{(by (2))}\\ &= \operatorname{Li}_2 \left (\frac{1}{\sqrt{2}} \right ) - \frac{\pi^2}{6} + \ln \left (\frac{1}{\sqrt{2}} \right ) \ln \left (1 - \frac{1}{\sqrt{2}} \right ) - \frac{1}{8} \ln^2 2 \qquad \text{(by (3))}\\ &= \operatorname{Li}_2 \left (\frac{1}{\sqrt{2}} \right ) - \frac{\pi^2}{6} -\frac{1}{2} \ln 2 \ln \left (\frac{1}{\sqrt{2}(1 + \sqrt{2})} \right ) - \frac{1}{8} \ln^2 2\\ &= \operatorname{Li}_2 \left (\frac{1}{\sqrt{2}} \right ) - \frac{\pi^2}{6} + \frac{1}{2} \ln 2 \left [\frac{1}{2} \ln 2 + \ln (1 + \sqrt{2}) \right ] - \frac{1}{8} \ln^2 2\\ &=\operatorname{Li}_2 \left (\frac{1}{\sqrt{2}} \right ) - \frac{\pi^2}{6} + \frac{1}{2} \ln 2 \ln (1 + \sqrt{2}) + \frac{1}{8} \ln^2 2. \end{align} Thus \begin{align} \operatorname{Li}_2 (\sqrt{2} - 1) - \operatorname{Li}_2 (1 - \sqrt{2}) &= - \left [\operatorname{Li}_2 \left (\frac{1}{\sqrt{2}} \right ) + \operatorname{Li}_2 \left (-\frac{1}{\sqrt{2}} \right ) \right ] +\frac{\pi^2}{6} - \frac{1}{2} \ln^2 (1 + \sqrt{2}) - \frac{1}{4} \ln^2 2\\ &= -\frac{1}{2} \operatorname{Li}_2 \left (\frac{1}{2} \right ) + \frac{\pi^2}{6} - \frac{1}{2} \ln^2(1 + \sqrt{2}) - \frac{1}{4} \ln^2 2 \qquad \text{(by (4))} \\ &= -\frac{1}{2} \left [\frac{\pi^2}{12} - \frac{1}{2} \ln^2 2 \right ] + \frac{\pi^2}{6} - \frac{1}{2} \ln^2(1 + \sqrt{2}) -\frac{1}{4} \ln^2 2\\ &= \frac{\pi^2}{8} - \frac{1}{2} \ln^2 (1 + \sqrt{2}). \end{align} Here the well-known value for the dilogarithm function at $x = 1/2$ has been used.

So the sum in ($\dagger$) can be expressed as $$\sum_{n = 0}^\infty \frac{(\sqrt{2} - 1)^{2n + 1}}{(2n + 1)^2} = \frac{\pi^2}{16} - \frac{1}{4} \ln^2 (1 + \sqrt{2}).$$

Putting the final pieces of the puzzle together, (1) becomes \begin{align} \int_{1}^{1 + \sqrt{2}} \frac{\ln x}{x^2 - 1} \, dx &= -\frac{1}{2} \ln^2 (1 + \sqrt{2}) + \frac{\pi^2}{8} - \left [\frac{\pi^2}{16} - \frac{1}{4} \ln^2 (1 + \sqrt{2}) \right ]\\ &= \frac{\pi^2}{16} - \frac{1}{4} \ln^2 (1 + \sqrt{2}), \end{align} as required.