A difficult integral evaluation problem

Since it hasn't been specifically objected to yet, here is a solution that doesn't rely on complex variable methods.

We shall make use of the Fourier sine series,

$$\frac{a\sin x}{1-2a\cos x+a^2}=\begin{cases} \sum_{n=1}^{\infty}a^{n}\sin{(nx)},~~~\text{for }|a|<1,\\ \sum_{n=1}^{\infty}\frac{\sin{(nx)}}{a^{n}},~~~\text{for }|a|>1. \end{cases}$$

These series may readily be derived by taking the imaginary parts of the complex geometric series $\sum_{n=1}^{\infty}\left(ae^{ix}\right)^n$ and $\sum_{n=1}^{\infty}\left(\frac{1}{a}e^{ix}\right)^n$, respectively.

By expanding the integrand in terms of these series and then swapping the order of integration and summation, a closed form may be obtained. For the $|a|>1$ case,

$$\begin{align}I(a)&=\int_0^\pi \frac{x\sin x}{1-2a\cos x+a^2}\mathrm{d}x\\ &=\int_{0}^{\pi}x\sum_{n=1}^{\infty}\frac{\sin{(nx)}}{a^{n+1}}\mathrm{d}x\\ &=\sum_{n=1}^{\infty}\frac{1}{a^{n+1}}\int_{0}^{\pi}x\sin{(nx)}\mathrm{d}x\\ &=\sum_{n=1}^{\infty}\frac{1}{a^{n+1}}\left(\frac{\sin{(n\pi)}}{n^2}-\frac{\pi\cos{(n\pi)}}{n}\right)\\ &=\pi\sum_{n=1}^{\infty}\frac{(-1)^{n+1}}{n\,a^{n+1}}\\ &=\frac{\pi}{a}\log{\frac{1+a}{a}}. \end{align}$$

For the $0<|a|<1$ cases, we can make use of the fact that if $0<|a|<1$, then $\frac{1}{|a|}>1$, thus allowing us to invoke the previous result. Hence,

$$\begin{align}I(a)&=\int_0^\pi \frac{x\sin x}{1-2a\cos x+a^2}\mathrm{d}x\\ &=\frac{1}{a^2}\int_0^\pi \frac{x\sin x}{a^{-2}-2a^{-1}\cos x+1}\mathrm{d}x\\ &=\frac{1}{a^2}\frac{\pi}{a^{-1}}\log{\frac{1+a^{-1}}{a^{-1}}}\\ &=\frac{\pi}{a}\log{(1+a)}. \end{align}$$

I'll first consider the case $0<a<1$.

Let $\displaystyle f(z) = \frac{z}{a-e^{-iz}}$ and integrate around a rectangle with vertices at $\pm \pi$ and $\pm \pi + iR$.

The function $f(z)$ has poles where $a-e^{-iz}=e^{\ln a + 2 \pi i n}-e^{-iz}= 0$.

That is, when $z= i \ln a - 2 \pi n$.

If $ 0<a< 1$, all of those points are in the lower half-plane.

Letting $ R \to \infty$ and going counterclockwise around the contour,

$$ \int_{- \pi}^{0} \frac{z}{a-e^{-ix}} \ dx + \int_{0}^{\pi} \frac{x}{a-e^{-ix}} \ dx + i \int_{0}^{\infty}f(\pi + i t) \ dt + \lim_{R \to \infty} \int_{-\pi}^{\pi} f(t + iR) \ dt $$

$$+ \ i \int_{\infty}^{0} f(-\pi+it) \ dy =0$$

Combing the first two integrals,

$$ \begin{align} \int_{- \pi}^{0} \frac{x}{a-e^{-ix}} \ dx + \int_{0}^{\pi} \frac{x}{a-e^{-ix}} \ dx &= -\int_{0}^{\pi} \frac{u}{a-e^{iu}} \ du + \int_{0}^{\pi} \frac{x}{a-e^{-ix}} \ dx \\ &= \int_{0}^{\pi} \frac{-x(a-e^{-ix})+ x(a-e^{ix}) }{(a-e^{ix})(a-e^{-ix})} \ dx \\ &= - 2 i \int_{0}^{\pi} \frac{x \sin x}{1-2a \cos x +a^{2}} \ dx \end{align}$$

Combining the third and fifth integrals,

$$ \begin{align} i \int_{0}^{\infty} f(\pi + it) \ dt - i \int_{0}^{\infty} f(-\pi + it) \ dt &= i \int_{0}^{\infty} \frac{\pi + it}{a-e^{-i(\pi + it)}} \ dt - i \int_{0}^{\infty} \frac{- \pi + it}{a-e^{-i( -\pi + it)}} \ dy \\ &= i \int_{0}^{\infty} \frac{\pi + it}{a +e^{t}} \ dt - i \int_{0}^{\infty} \frac{- \pi + it}{a+ e^{t}} \ dt \\ &= 2 \pi i \int_{0}^{\infty} \frac{1}{a+e^{t}} \ dt \\ &= 2 \pi i \int_{0}^{\infty} \frac{e^{-t}}{1+ae^{-t}} \ dt \\ &= - \frac{2 \pi i}{a} \ln (1+ae^{-t}) \Big|^{\infty}_{0} \\ &= \frac{2 \pi i}{a} \ln(1+a) \end{align}$$

And the fourth integral vanishes.

So we have

$$ - 2i \int_{0}^{\pi} \frac{x \sin x}{1-2a \cos x +a^{2}} \ dx + \frac{2 \pi i}{a} \ln(1+a) = 0$$

or

$$ \int_{0}^{\pi } \frac{x \sin x}{1-2a \cos x +a^{2}} \ dx = \frac{\pi \ln (1+a)}{a}$$

The case for when $a >1$ is similar.

The only difference is that there is now a pole inside of the contour at $z=i \ln a$ with residue $ \displaystyle \frac{\ln a}{a}$.

And therefore,

$$- 2i \int_{0}^{\pi} \frac{x \sin x}{1-2a \cos x +a^{2}} \ dx + \frac{2 \pi i}{a} \ln(1+a) = 2 \pi i \frac{\ln a}{a} $$

or

$$ \int_{0}^{\pi} \frac{x \sin x}{1-2a \cos x+a^{2}} \ dx = \frac{\pi \ln (1+a)}{a} - \frac{\pi \ln a}{a} = \frac{\pi \ln \left(\frac{1+a}{a} \right)}{a}$$

$\newcommand{\+}{^{\dagger}} \newcommand{\angles}[1]{\left\langle\, #1 \,\right\rangle} \newcommand{\braces}[1]{\left\lbrace\, #1 \,\right\rbrace} \newcommand{\bracks}[1]{\left\lbrack\, #1 \,\right\rbrack} \newcommand{\ceil}[1]{\,\left\lceil\, #1 \,\right\rceil\,} \newcommand{\dd}{{\rm d}} \newcommand{\down}{\downarrow} \newcommand{\ds}[1]{\displaystyle{#1}} \newcommand{\expo}[1]{\,{\rm e}^{#1}\,} \newcommand{\fermi}{\,{\rm f}} \newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,} \newcommand{\half}{{1 \over 2}} \newcommand{\ic}{{\rm i}} \newcommand{\iff}{\Longleftrightarrow} \newcommand{\imp}{\Longrightarrow} \newcommand{\isdiv}{\,\left.\right\vert\,} \newcommand{\ket}[1]{\left\vert #1\right\rangle} \newcommand{\ol}[1]{\overline{#1}} \newcommand{\pars}[1]{\left(\, #1 \,\right)} \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}} \newcommand{\pp}{{\cal P}} \newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,} \newcommand{\sech}{\,{\rm sech}} \newcommand{\sgn}{\,{\rm sgn}} \newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}} \newcommand{\ul}[1]{\underline{#1}} \newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert} \newcommand{\wt}[1]{\widetilde{#1}}$ $\ds{\int_{0}^{\pi}{x\sin\pars{x} \over 1-2a\cos\pars{x} + a^{2}}\,\dd x:\ {\large ?}.\qquad{\large a > 0}}$.

\begin{align}&\color{#c00000}{\int_{0}^{\pi} {x\sin\pars{x} \over 1-2a\cos\pars{x} + a^{2}}\,\dd x} =\half\int_{-\pi}^{\pi}{x\sin\pars{x} \over \pars{a - \expo{\ic x}}\pars{a - \expo{-\ic x}}}\,\dd x \\[3mm]&=\half\int_{-\pi}^{\pi}{x\sin\pars{x} \over 2\ic\sin\pars{x}} \,\pars{{1 \over a - \expo{\ic x}} -{1 \over a - \expo{-\ic x}}}\,\dd x =\half\,\Im\int_{-\pi}^{\pi}{x \over a - \expo{\ic x}}\,\dd x \\[3mm]&=\half\Im \int_{\verts{z}\ =\ 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}} {-\ic\ln\pars{z} \over a - z}\,{\dd z \over \ic z} \\[3mm]&={1 \over 2a}\,\Im\bracks{% \int_{\verts{z}\ =\ 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}} {\ln\pars{z} \over z - a}\,\dd z\ -\ \overbrace{% \int_{\verts{z}\ =\ 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}} {\ln\pars{z} \over z}\,\dd z}^{\ds{=\ 0}}} \end{align}

$$ \color{#c00000}{\int_{0}^{\pi} {x\sin\pars{x} \over 1-2a\cos\pars{x} + a^{2}}\,\dd x} ={1 \over 2a}\,\Im \int_{\verts{z}\ =\ 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}} {\ln\pars{z} \over z - a}\,\dd z $$

\begin{align}&\color{#c00000}{\int_{0}^{\pi} {x\sin\pars{x} \over 1-2a\cos\pars{x} + a^{2}}\,\dd x} \\[3mm]&={1 \over 2a}\Im\bracks{% 2\pi\ic\ln\pars{a}\Theta\pars{1 - a} -\int_{-1}^{0}{\ln\pars{-x} + \ic\pi \over x - a}\,\dd x -\int_{0}^{-1}{\ln\pars{-x} - \ic\pi \over x - a}\,\dd x} \\[3mm]&={\pi \over a}\bracks{\ln\pars{a}\Theta\pars{1 - a} -\int_{-1}^{0}{\dd x \over x - a}} ={\pi \over a}\bracks{\ln\pars{a}\Theta\pars{1 - a} + \ln\pars{1 + a \over a}} \end{align}

$$\color{#66f}{\large% \left.\int_{0}^{\pi}{x\sin\pars{x}\,\dd x \over 1-2a\cos\pars{x} + a^{2}} \right\vert_{\ds{\color{#c00000}{\,a > 0}}}} =\color{#66f}{\large% \left\lbrace\begin{array}{lcrcl} {\pi \over a}\,\ln\pars{1 + a} & \mbox{if} & a & < & 1 \\[3mm] {\pi \over a}\,\ln\pars{1 + a \over a} & \mbox{if} & a & > & 1 \end{array}\right.} $$