Compute $\int_0^\pi\frac{\cos nx}{a^2-2ab\cos x+b^2}\, dx$

Let $I_n(a,b)$ be the desired integral. Note that $I_n(a,b)=I_n(b,a)$, and $I_n(a,b)=I_n(-a,-b)$. So, we may suppose that $|b|< a$ Note that $$\eqalign{ \frac{a^2-b^2}{a^2-2ab\cos x+b^2}&=\frac{a}{a-e^{ix}b}+\frac{be^{-ix}}{a-e^{-ix}b}\cr &=\sum_{n=0}^\infty \left(\frac{b}{a}\right)^ne^{inx}+\frac{be^{-ix}}{a}\sum_{n=0}^\infty \left(\frac{b}{a}\right)^ne^{-inx}\cr &=1+\sum_{n=1}^\infty \left(\frac{b}{a}\right)^ne^{inx}+ \sum_{n=1}^{\infty} \left(\frac{b}{a}\right)^{n}e^{-inx}\cr &=1+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\cos(n x) } $$ It follows, using the uniform convergence of the series on $[0,\pi]$, that $$ \int_0^\pi\frac{(a^2-b^2)\cos(mx)}{a^2-2ab\cos x+b^2}dx =\int_0^\pi\cos(mx)dx+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\int_0^\pi\cos(n x)\cos(mx)dx $$ But $\int_0^\pi\cos(n x)\cos(mx)dx=0$ if $n\ne m$, and $\int_0^\pi\cos^2(n x)dx=\pi/2$ if $n\ne0$. So $$\eqalign{I_m(a,b)= \int_0^\pi\frac{\cos(mx)}{a^2-2ab\cos x+b^2}dx &=\left\{\matrix{\frac{\pi}{a^2-b^2}&\hbox{if}&m=0\cr \frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^m&\hbox{if}&m\ne0 } \right.} $$ which is the desired formula for $|b|<a$.

Assume for definiteness that $a>b>0$.

Method 1:

For integer $n\geq0$, we can rewrite the integral as $$\frac{1}{4ab}\int_{-\pi}^{\pi}\frac{e^{in x}dx}{\cosh\gamma-\cos x}=\frac{1}{2ia^2}\oint_{|z|=1}\frac{z^{n}dz}{(z-e^{-\gamma})(1-e^{-\gamma}z)},$$ where $e^{\gamma}=\frac{a}{b}$. Computing the residue at $z=e^{-\gamma}$, we find for the last integral $$2\pi i \cdot \frac{1}{2ia^2}\cdot\frac{\left(\frac{b}{a}\right)^n}{1-\frac{b^2}{a^2}}=\frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^n.$$

Method 2:

Similarly rewrite the integral as $$\frac{1}{2(a^2-b^2)}\int_{-\pi}^{\pi}\left(\frac{1}{1-\frac{b}{a}e^{ix}}+\frac{\frac{b}{a}e^{-ix}}{1-\frac{b}{a}e^{-ix}}\right)e^{in x}dx.$$ Then expand the integrand into series in $\frac{b}{a}$ and use that $\displaystyle\int_{-\pi}^{\pi}e^{inx}dx=2\pi\cdot\delta_{n,0}$.

I don't have a proof yet, but with some computer assistance, this is what I believe to be the answer when $n$ is a non-negative integer.

\begin{align*} \int_0^\pi\frac{\cos nx}{a^2-2ab\cos x+b^2}\, dx &= \frac{\displaystyle \left(\left(a^{2n}+b^{2n}\right)\, \left(\Big\lvert \frac{a+b}{a-b}\Big\rvert +1\right)-2\, \sum_{k=0}^{2n}a^k\, b^{2n-k}\right)\, \pi}{\displaystyle 2\, a^n\, b^n\, (a+b)^2} \end{align*}

I think there might be a reduction formula for this, but at the moment I don't know.