Calculate this integral $\int_a^b (\int_a^b \frac{f(t) \overline{f(s)}}{1-ts} \,ds) \, dt$

You're almost there. You have for $|a|<1$ and $|b|<1$

$$\int_a^b\int_a^b \frac{f(t)\overline{f(s)}}{1-ts}\,ds\,dt= \int_a^b\int_a^b \sum_{n=0}^\infty f(t)t^n\overline{f(s)}s^n\,ds\,dt$$

Next, note that

$$\begin{align} \lim_{N\to\infty}\int_a^b\int_a^b f(t)\overline{f(s)}\,\left(\frac{1-(ts)^{N+1}}{1-ts}\right)\,ds\,dt&=\sum_{n=0}^\infty \int_a^b\int_a^b f(t)t^n\overline{f(s)}s^n\,ds\,dt\\\\ &=\sum_{n=0}^\infty \left(\int_a^b f(t)t^n\,dt\right)\left(\overline{\int_a^b f(t)t^n\,dt}\right)\\\\ &=\sum_{n=0}^\infty \left|\int_a^b f(t)t^n\,dt\right|^2 \end{align}$$

Since $f$ is continuous, then its magnitude is bounded and the Dominated Convergence Theorem guarantees that we can pass the limit under the integral to arrive at

$$\int_a^b\int_a^b \frac{f(t)\overline{f(s)}}{1-ts}\,ds\,dt=\sum_{n=0}^\infty \left|\int_a^b f(t)t^n\,dt\right|^2$$

as was to be shown!