Show that $\int_0^1\frac{\ln(1+x)}{1+x^2}\mathrm dx=\frac{\pi}8\ln 2$

I suppose what you could do is write $$\log (1 + \tan y) = \frac{1}{2} \log 2 + \log \sin (y + \tfrac{\pi}{4} ) - \log \cos y,$$ then transform the second term with $v = \pi/4 - y$ to obtain $$\int_{y=0}^{\pi/4} \log \sin (y + \tfrac{\pi}{4}) \, dy = \int_{v=\pi/4}^0 \log \sin (\tfrac{\pi}{2} - v) \, (-dv) = \int_{v=0}^{\pi/4} \log \cos v \, dv.$$ Then this cancels with the integral of the third term, and you are left with $$\frac{\pi}{8} \log 2$$ as claimed.

Here's a different solution.

Let

$$f=\int_0^1 \frac{\log (x+1)}{x^2+1} \, dx$$

Replacing the $\log$ by the defining integral in the form

$$\log (x+1)=\int_0^1 \frac{x}{x y+1} \, dy$$

we find a double integral form of $f$

$$f = \int _0^1\int _0^1\frac{x}{\left(x^2+1\right) (x y+1)}dydx$$

Exchanging now the order of integration, i.e. carrying out the x-integration first, which is elementary, gives

$$\int_0^1 \frac{x}{\left(x^2+1\right) (x y+1)} \, dx=\frac{\pi y}{4 y^2+4}+\frac{\log (4)}{4 y^2+4}-\frac{\log (y+1)}{y^2+1}$$

The integral over y now gives us the negative of the original integral f and two further terms which can be integrated to give

$$\int_0^1 \left(\frac{\pi y}{4 y^2+4}+\frac{\log (4)}{4 y^2+4}\right) \, dy=\frac{1}{4} \pi \log (2)$$

Hence we find that

$$f=\frac{1}{4} \pi \log (2)-f$$

or

$$f=\frac{1}{8} \pi \log (2)$$

QED.

Without using the given hint use the substitution $u = \frac{\pi}{4} -x$ to get $$I = \int_0^{\pi/4} \log (1 + \tan x) \, \mathrm{d}x = \int_0^{\pi/4} \log \left(\frac{2}{1 + \tan u}\right) \, \mathrm{d}u = \frac{\pi}{4}\ln 2 - I$$ since $1 + \tan (\pi/4 - u) = 1 + \frac{1-\tan u}{1 + \tan u} = \frac{2}{1 + \tan u}$.

So $2I = \frac{\pi}{4}\ln 2 \iff I = \frac{\pi}{8}\ln 2$