Evaluate $\int_0^{\pi} \frac{\sin^2 \theta}{(1-2a\cos\theta+a^2)(1-2b\cos\theta+b^2)}\mathrm{d\theta}, \space 0<a<b<1$

This integral is $1/2$ the integral over $[0,2 \pi)$. Let $z=e^{i \theta}$, $d\theta = dz/(i z)$; the result is

$$\frac{1}{2 i}\oint_{|z|=1} \frac{dz}{z} \frac{-\frac{1}{4} (z^2-1)^2}{(a z^2-(1+a^2)z+a)(b z^2-(1+b^2)z+b)}$$

which can be rewritten as

$$\frac{i}{8} \oint_{|z|=1} \frac{dz}{z} \frac{(z^2-1)^2}{(a z-1)(z-a)(b z-1)(z-b)}$$

There are 5 poles, although because $0<a<b<1$, only 3 of them fall within the contour. This integral is then $i 2 \pi$ times the sum of the residues of these poles. The residues of these poles are actually straightforward:

$$\mathrm{Res}_{z=0}\frac{i}{8} \frac{(z^2-1)^2}{z(a z-1)(z-a)(b z-1)(z-b)} = \frac{i}{8 a b}$$ $$\mathrm{Res}_{z=a}\frac{i}{8} \frac{(z^2-1)^2}{z(a z-1)(z-a)(b z-1)(z-b)} = \frac{i}{8 a} \frac{a^2-1}{(a b-1)(a-b)}$$ $$\mathrm{Res}_{z=b}\frac{i}{8} \frac{(z^2-1)^2}{z(a z-1)(z-a)(b z-1)(z-b)} = -\frac{i}{8 b} \frac{b^2-1}{(a b-1)(a-b)}$$

There is vast simplification from adding these pieces together, which I leave to the reader. The result is

$$\int_0^{\pi} d\theta \frac{\sin^2 \theta}{(1-2a\cos\theta+a^2)(1-2b\cos\theta+b^2)} = \frac{\pi}{2} \frac{1}{1-a b}$$

It can be done easily without complex, if we note that $$ \frac{\sin x}{1-2a\cos x+a^2}=\sum_{n=0}^{+\infty}a^n\sin[(n+1)x]$$

Just saying.

EDIT: for proving this formula, we actually use complex method

Using the series presented by Cortizol, the integral can be written as: $$\int_0^{\pi} dx\left(\sum_{n=1}^{\infty} a^{n-1}\sin (nx)\right)\left(\sum_{m=1}^{\infty} b^{m-1}\sin (mx)\right)=\frac{1}{ab}\sum_{n=1}^{\infty} \sum_{m=1}^{\infty} a^nb^m\int_0^{\pi} \sin(nx)\sin(mx)\,dx $$ Notice that for $n\ne m$, the integral is always zero, hence we look at only those cases where $n=m$ i.e $$\frac{1}{ab}\sum_{n=1}^{\infty} (ab)^n\int_0^{\pi} \sin^2(nx)\,dx=\frac{\pi}{2ab}\sum_{n=1}^{\infty} (ab)^n=\boxed{\dfrac{\pi}{2}\dfrac{1}{1-ab}}$$ $\blacksquare$