Proof of $\int_0^\infty \frac{x^{\alpha}dx}{1+2x\cos\beta +x^{2}}=\frac{\pi\sin (\alpha\beta)}{\sin (\alpha\pi)\sin \beta }$

If you don't mind, I would like to present an alternative approach that makes use of the fact that $$\int^\infty_0\frac{x^{p-1}}{1+x}dx=\frac{\pi}{\sin{p\pi}}$$ Simply factorise the denominator and decompose the integrand into partial fractions. \begin{align} \int^\infty_0\frac{x^a}{x^2+2(\cos{b})x+1}dx &=\int^\infty_0\frac{x^a}{(x+e^{ib})(x+e^{-ib})}dx\\ &=\frac{1}{-e^{ib}+e^{-ib}}\int^\infty_0\frac{x^a}{e^{ib}+x}dx+\frac{1}{-e^{-ib}+e^{ib}}\int^\infty_0\frac{x^a}{e^{-ib}+x}dx\\ &=\frac{1}{-2i\sin{b}}\int^\infty_0\frac{(e^{ib}u)^a}{1+u}du+\frac{1}{2i\sin{b}}\int^\infty_0\frac{(e^{-ib}u)^a}{1+u}du\\ &=\frac{e^{iab}}{-2i\sin{b}}\frac{\pi}{\sin(\pi a+\pi)}+\frac{e^{-iab}}{2i\sin{b}}\frac{\pi}{\sin(\pi a+\pi)}\\ &=\frac{\pi}{\sin \pi a\sin{b}}\left(\frac{e^{iab}-e^{-iab}}{2i}\right)\\ &=\frac{\pi\sin{ab}}{\sin{\pi a}\sin{b}} \end{align}

Step 1. Divide $$I=\int_0^\infty=\int_0^1+\int_1^\infty$$ and in the second integral apply the substitution $x=1/t$: $$I=\int_0^1\frac{x^\alpha \,dx}{1+2x\cos\beta+x^2}+\int_0^1\frac{t^{-\alpha} \,dt}{t^2+2t\cos\beta+1}=\int_0^1\frac{(x^\alpha+x^{-\alpha}) \,dx}{1+2x\cos\beta+x^2}.$$

Step 2. Use the following series expansion: $$\sum_{n=1}^\infty (-1)^{n+1}x^n\sin n\beta=\frac{x\sin\beta}{1+2x\cos\beta+x^2},\quad |x|<1$$ (it is easily obtained by considering imaginary part of $\sum\limits_{n=1}^\infty q^n=\frac{q}{1-q}$ with $q=-xe^{i\beta}$). Integrating term-by-term yields $$I=\frac{1}{\sin\beta}\int_0^1\sum_{n=1}^\infty (-1)^{n+1}(x^{n-1+\alpha}+x^{n-1-\alpha})\sin n\beta\,dx=\frac{1}{\sin\beta}\sum_{n=1}^\infty\frac{(-1)^{n+1}2n\sin n\beta}{n^2-\alpha^2}.$$

Step 3. Recall Fourier series expansion for the function $\sin\alpha x$: $$\sin\alpha x=\frac{\sin\pi\alpha}{\pi}\sum_{n=1}^\infty\frac{(-1)^{n+1}2n\sin nx}{n^2-\alpha^2},\quad x\in(-\pi,\pi),$$ which is proved by calculating of Fourier coefficients.

Taking here $x=\beta$ we conclude that $$I=\frac{\pi\sin\alpha\beta}{\sin\pi\alpha\sin\beta}.$$

If I may use contours to show this one.

Consider $\displaystyle I=\frac{z^{a}}{z^{2}-2z\cos(b)+1}, \;\ 0<a<1, \;\ 0<b<\pi$

Use a semicircle in the UHP with center at origin.

The poles lie at $z=e^{\pm ib}=\cos(b)\pm i\sin(b)$

Along the x axis:

$$\int_{-\infty}^{0}\frac{x^{a}}{x^{2}-2x\cos(b)+1}dx+\int_{0}^{\infty}\frac{x^{a}}{x^{2}-2x\cos(b)+1}dx$$

Let $x\to -x$ in the first integral and get:

$$\int_{0}^{\infty}\frac{x^{a}}{x^{2}-2x\cos(b)+1}dx+\int_{0}^{\infty}\frac{(-1)^{a}x^{a}}{x^{2}+2x\cos(b)+1}dx, \;\ e^{\pi ia}=(-1)^{a}$$

Round the big arc:

$$\int_{0}^{\pi}\frac{R^{a}e^{ia\theta}iRe^{i\theta}}{R^{2}e^{2i\theta}-2Re^{i\theta}\cos(b)+1}d\theta$$

which, since $a<1$, vanishes as $R\to 0$

By using $z=e^{ib}+re^{i\phi}$, the residue at $e^{ib}$ is

$$2\pi i Res(e^{ib})=2\pi i \lim_{r\to 0}\frac{(e^{ib}+re^{i\phi})^{a}}{e^{ib}+re^{i\phi}-e^{-ib}}$$

$$=\frac{2\pi i e^{iab}}{e^{ib}-e^{-ib}}=\frac{\pi}{\sin(b)}e^{iab}$$

Thus,

$$\int_{0}^{\infty}\frac{x^{a}}{x^{2}-2x\cos(b)+1}dx+e^{i\pi a}\int_{0}^{\infty}\frac{x^{a}}{x^{2}+2x\cos(b)+1}dx=\frac{\pi}{\sin(b)}e^{iab}$$

Equate real and imaginary parts:

$$\int_{0}^{\infty}\frac{x^{a}}{x^{2}-2x\cos(b)+1}dx+\cos(\pi a)\int_{0}^{\infty}\frac{x^{a}}{x^{2}+2x\cos(b)+1}dx=\frac{\pi}{\sin(b)}\cos(ab)$$

and $$\sin(\pi a)\int_{0}^{\infty}\frac{x^{a}}{x^{2}+2x\cos(b)+1}dx=\frac{\pi}{\sin(b)}\sin(ab)$$

Hence, we have a two for one:

$$\int_{0}^{\infty}\frac{x^{a}}{x^{2}+2x\cos(b)+1}dx=\frac{\pi}{\sin(b)}\frac{\sin(ab)}{\sin(\pi a)}$$ and

$$\int_{0}^{\infty}\frac{x^{a}}{x^{2}-2x\cos(b)+1}dx=\frac{\pi}{\sin(b)}\frac{\sin(a(\pi -b))}{\sin(\pi a)}$$

The last one follows from the first from writing $\pi -b$ for $b$.