How to evaluate $\int_{0}^{\infty} \frac{x^{-\mathfrak{i}a}}{x^2+bx+1} \,\mathrm{d}x$ using complex analysis?
$\phantom{}$ Dear MSE users, this is the new episode of Mister Feynman and Monsieur Laplace versus contour integration.
Tonight we have a scary integral, but we may immediately notice that
$$ \mathcal{L}(x^{-ia})(s) = s^{ia-1}\Gamma(1-ia),\qquad \mathcal{L}^{-1}\left(\frac{1}{x^2+bx+1}\right)(s) =\frac{e^{Bs}-e^{\overline{B}s}}{\sqrt{b^2-4}}$$
where $B$ is the root of $x^2+bx+1$ with a positive imaginary part. By the properties of the Laplace transform, the original integral is converted into
$$\frac{\Gamma(1-ia)}{\sqrt{b^2-4}}\int_{0}^{+\infty}s^{ia-1}\left(e^{Bs}-e^{\overline{B}s}\right)\,ds$$
which can be evaluated in terms of the $\Gamma$ function.
Due to the reflection formula, the final outcome simplifies into
$$ \frac{\left(B^{-i a}-\overline{B}^{-i a}\right) \pi }{\sinh(\pi a)\sqrt{4-b^2}} $$
and we may notice that $B=\exp\left[i\arccos\frac{b}{2}\right]$ allows a further simplification.
Outro: poor children.
$\newcommand{\Res}{\text{Res}}$ Firstly define: \begin{align} f(z) = \frac{1}{z^2+bz+1} \end{align} Secondly define: \begin{align} g(z)= (-z)^{-ia}f(z) \end{align} We use the Principal Log to define $(-z)^{-ia}$. The main reason for the minus sign is that I want to work with the principal Log. Consider now the keyhole contour $K_R$ consisting of $C_R \cup C_R^+ \cup C_R^-$. The circle part is $C_R$ with radius $R$, and $C_R^+$ is the segment that connects $0$ to $R$ on the right half plane from above and $C_R^-$ the same as $C_R^+$ but from below.
So our contour looks like the one below:
Note that $\int_{C_R} g(z)dz \to 0$ as $R\to \infty$ (why?). Now the poles of this function $g(z)$ is at $z_1= -\frac{1}{2}b - i\ \sqrt[]{1-\frac{1}{4}b^2}$ and $z_2 = -\frac{1}{2}b + i\ \sqrt[]{1-\frac{1}{4}b^2}$. Note that for large enough $R$ both poles will be in the area enclosed by our contour.
On $C_R^+$ we have: \begin{align} \lim_{R\to \infty} \int_{C_R^+} g(z) dz = e^{-a\pi}\int^{\infty}_0\frac{x^{-ia}}{x^2+bx+1} dx \end{align} Similarly on $C_R^-$ we have: \begin{align} \lim_{R\to \infty} \int_{C_R^-} g(z) dz = -e^{a\pi}\int^{\infty}_0\frac{x^{-ia}}{x^2+bx+1} dx \end{align} So: \begin{align} \lim_{R\to\infty} \int_{K_R} g(z) dz &= (-e^{a\pi} + e^{-a\pi} )\int^\infty_0 \frac{x^{-ia}}{x^2+bx+1} dx \\ &=-2\sinh(a\pi) \int^\infty_0 \frac{x^{-ia}}{x^2+bx+1} dx \end{align} On the other hand we have by the Residue theorem: \begin{align} \lim_{R\to\infty} \int_{K_R} g(z) dz = 2\pi i\left( \Res_{z=z_1}g(z)+\Res_{z=z_2}g(z)\right) \end{align} Let's calculate the residues: \begin{align} \Res_{z=z_1} g(z) = \frac{\left(\frac{1}{2}b + i\ \sqrt[]{1-\frac{1}{4}b^2}\ \right)^{-ia}}{-2i \ \sqrt[]{1-\frac{1}{4}b^2} } \end{align} The other one: \begin{align} \Res_{z=z_2} g(z) = \frac{\left(\frac{1}{2}b - i\ \sqrt[]{1-\frac{1}{4}b^2}\ \right)^{-ia}}{2i \ \sqrt[]{1-\frac{1}{4}b^2} } \end{align} Define $\beta:=\frac{1}{2}b + i\ \sqrt[]{1-\frac{1}{4}b^2}$. So: \begin{align} \lim_{R\to\infty} \int_{K_R} g(z) dz = \pi \frac{\bar\beta^{-ia}-\beta^{-ia}}{\sqrt[]{1-\frac{1}{4}b^2}} = 2\pi \frac{\bar\beta^{-ia}-\beta^{-ia}}{ \sqrt[]{4-b^2}} \end{align} This means: \begin{align} \int^{\infty}_0 \frac{x^{-ia}}{x^2+bx+1}dx &= \frac{2\pi}{-2\sinh(a\pi)} \frac{\bar\beta^{-ia}-\beta^{-ia}}{ \sqrt[]{4-b^2}} \end{align} Note that $\beta = \exp(i\arccos(b/2))$. So $\beta^{-ia}=\exp(a\arccos(b/2))$ and $\bar\beta^{-ia}=\exp(a\arccos(b/2)).$ Substituting $\beta$ gives us the final result:
\begin{align} \int^{\infty}_0 \frac{x^{-ia}}{x^2+bx+1}dx = \color{red}{\frac{2\pi\sinh(a\cdot\arccos(b/2))}{\sinh(a\pi) \ \sqrt[]{4-b^2}} } \end{align}
This integral is real.
Remark
It may be interesting to note the following. For positive $x$ one has: $x^{-ia}=e^{-ia\text{Log}(x)}=e^{-ia\ln(x)}=\cos(a\ln(x))-i\sin(a\ln(x))$. So we have: \begin{align} \int^\infty_0 \frac{x^{-ia}}{x^2+bx+1} dx = \int^\infty_0 \frac{\cos(a\ln(x))}{x^2+bx+1} dx - i\int^\infty_0 \frac{\sin(a\ln(x))}{x^2+bx+1} dx \end{align} Since our integral is real we get the following identities for free: \begin{align} \color{blue}{\int^\infty_0 \frac{\cos(a\ln(x))}{x^2+bx+1} dx = \frac{2\pi\sinh(a\cdot\arccos(b/2))}{\sinh(a\pi) \ \sqrt[]{4-b^2}}} \end{align} And: \begin{align} \color{blue}{\int^\infty_0 \frac{\sin(a\ln(x))}{x^2+bx+1} dx = 0 } \end{align}
In a complex analysis class, you’re more likely to come across the integral $$\int_{0}^{\infty} \frac{x^{\alpha}}{1+2x \cos \beta +x^{2}} \, dx = \frac{\pi \sin (\alpha \beta)}{\sin(\alpha \pi) \sin(\beta)}, \quad (-1<\alpha <1, \ 0 < \beta < \pi).$$
See this answer, for example, which uses a semicircle in the upper half-plane that is indented at the origin.
Alternatively, you could also use the keyhole contour that Shashi used.
Also see this question about the peculiar symmetry of this integral when it's written in a slightly different form.
The function on the right is not defined at $\alpha =0$. But you could show separately (by using the same contour or by completing the square) that the value of the integral at $\alpha =0$ is $$\lim_{a \to 0} \frac{\pi \sin (\alpha \beta)}{\sin(\alpha \pi) \sin(\beta)} = \frac{\beta}{\sin \beta}.$$
Now if we assign the function on the right the value $\frac{\beta}{\sin \beta}$ at $\alpha =0$, then right side of the equation is a holomorphic function for $-1 <\operatorname{Re}(\alpha) <1$ with $\beta$ fixed.
And since the integral on the left is absolutely convergent in the strip $-1 < \operatorname{Re}(\alpha) < 1$, we can use a property of the Mellin transform that states that the integral defines an holomorphic function in that strip.
(This is very similar to property of the Laplace transform mentioned here, and can be proved in essentially the same manner.)
So by the identity theorem, the formula holds for $-1 <\operatorname{Re}(\alpha) <1$.
Your integral is the case $\alpha=-ia$ and $\cos(\beta) = \frac{b}{2}$.
If $b\in [0, 2)$, then $\beta$ falls between means $0$ and $\pi$, and we get
$$\begin{align} \int_{0}^{\infty} \frac{x^{-ia}}{x^{2}+bx+1} \, dx &= \frac{\pi \sin\left(-ia \arccos\left(\frac{b}{2} \right)\right)}{\sin(-ia \pi) \sin \left(\arccos \left(\frac{b}{2} \right) \right)} \\ &= \frac{\pi \sinh \left(a \arccos\left(\frac{b}{2} \right) \right)}{\sinh(a \pi)\frac{\sqrt{4-b^{2}}}{2}}. \end{align} $$