A generalization for Serret's integral $\int_0^a \frac{\ln(1+ax)}{1+x^2}dx$
$$\begin{align}I=\int_0^a\frac{\ln(1+ax)}{1+x^2}dx&=\int_0^a\frac{\ln\frac{1+a^2}{1+at}}{1+t^2}dt\text{ (transform $x=\frac{a-t}{1+at}$)}\\ &=\arctan a\ln(1+a^2)-\int_0^a\frac{\ln(1+at)}{1+t^2}dt\\ &=\arctan a\ln(1+a^2)-I\\ &=\frac12\arctan a\ln(1+a^2)\end{align} $$ The key point is the mobius transformation.
How do I discover this transformation? By observing the similar technique OP mentioned. We may let $$x=\frac{b_1+b_2t}{c_1+c_2t}$$ and add some restrictions to $b_1,b_2,c_1,c_2$. We hope this transformation preserves the bounds of integral, hence $$0=\frac{b_1+b_2a}{c_1+c_2a}$$ and $$a=\frac{b_1}{c_1}.$$ Also, we can cancel out $b_2$. Three relations are found, then just try to substitute it into the original integral and hope it works. The result is that it really works.
Claim: for any $a>0$, $$\boxed{ I(a)=\int_{0}^{a}\frac{\log(1+ax)}{1+x^2}\,dx = \color{red}{\arctan(a)\log\sqrt{a^2+1}}.} \tag{0}$$ Proof: $$ I(a) = \int_{0}^{1}\frac{a\log(1+a^2 x)}{1+a^2 x^2}\,dx,$$ $$I'(a) = \color{blue}{\int_{0}^{1}\frac{2a^2 x}{(1+a^2 x)(1+a^2 x^2)}\,dx} - \color{purple}{ \int_{0}^{1}\frac{(1-a^2 x^2)}{(1+a^2 x^2)^2}\log(1+a^2 x)\,dx }\tag{1}$$ The blue integral is elementary and the purple integral turns into an elementary integral by integration by parts. In particular, the derivatives of both sides of $(0)$ match. To finish the proof, we just have to show that $(0)$ holds at $a=1$. But that is equivalent to $$ \int_{0}^{\pi/4}\log(1+\tan\theta)\,d\theta = \frac{\pi}{8}\log(2) \tag{2}$$ that is well-known. Another possible approach is to read the LHS of $(0)$ as $$ \int_{0}^{a}\frac{\log(1+ax)}{x}\cdot\frac{x}{x^2+1}\,dx $$ and exploit the dilogarithm reflection formulas.
Well, we know that:
$$1+x^2=\left(x-i\right)\left(x+i\right)\tag1$$
So, perform partial fraction decomposition:
$$\mathcal{I}\left(x\right):=\int\frac{\ln\left(1+\text{a}\cdot x\right)}{1+x^2}\space\text{d}x=\frac{i}{2}\cdot\left\{\int\frac{\ln\left(1+\text{a}\cdot x\right)}{x+i}\space\text{d}x-\int\frac{\ln\left(1+\text{a}\cdot x\right)}{x-i}\space\text{d}x\right\}\tag2$$
Now, substitute $\text{u}=x+i$ for the left integral:
$$\int\frac{\ln\left(1+\text{a}\cdot x\right)}{x+i}\space\text{d}x=\int\frac{\ln\left(1+\frac{\text{a}\cdot\text{u}}{1-\text{a}i}\right)}{\text{u}}\space\text{d}\text{u}+\ln\left(1-\text{a}i\right)\int\frac{1}{\text{u}}\space\text{d}\text{u}\tag3$$
Now, substitute $\text{v}=\frac{\text{a}\cdot\text{u}}{\text{a}i-1}$ for the left integral:
$$\int\frac{\ln\left(1+\frac{\text{a}\cdot\text{u}}{1-\text{a}i}\right)}{\text{u}}\space\text{d}\text{u}=-\int-\frac{\ln\left(1-\text{v}\right)}{\text{v}}\space\text{d}\text{v}:=\text{C}_1-\text{Li}_2\left(\text{v}\right)\tag4$$
And for the right integral:
$$\ln\left(1-\text{a}i\right)\int\frac{1}{\text{u}}\space\text{d}\text{u}=\ln\left(1-\text{a}i\right)\ln\left|\text{u}\right|+\text{C}_2\tag5$$
Now, for $\int\frac{\ln\left(1+\text{a}\cdot x\right)}{x-i}\space\text{d}x$ it is the same way of proceeding.