For $\alpha>0$, evaluate $\int^{+\infty}_0xe^{-x}\cos x\cos(x^2/\alpha)\,dx$

I will write

$$ J_{\pm} = \int_{0}^{\infty} xe^{-x}\cos\left(x\pm\frac{x^2}{\alpha}\right)\,\mathrm{d}x $$

so that your integral takes the form $\frac{1}{2}(J_{+} + J_{-})$. Then

\begin{align*} J_{\pm} &= \operatorname{Re}\left[ \int_{0}^{\infty} x\exp\left( -x + ix \pm \frac{ix^2}{\alpha}\right) \,\mathrm{d}x. \right] \end{align*}

Now write $\mathbb{H}_{\text{right}} = \{ z \in \mathbb{C} : \operatorname{Re}(z) > 0 \}$. Then for each $a \in \mathbb{H}_{\text{right}}$, the map $z \mapsto \int_{0}^{\infty} x e^{-ax-zx^2} \, \mathrm{d}x$ is analytic on $\mathbb{H}_{\text{right}}$ and continuous on $\overline{\mathbb{H}_{\text{right}}}$. Moreover, if $a, z \in (0, \infty)$, then with $b = a^2/4z$,

\begin{align*} \int_{0}^{\infty} x e^{-ax-zx^2} \, \mathrm{d}x &= \int_{0}^{\infty} x \exp\bigg( -b \left(\frac{2z x}{a}+1\right)^2 + b \bigg) \, \mathrm{d}x \\ &= \frac{be^{b}}{z} \int_{1}^{\infty} (u-1) e^{-bu^2} \, \mathrm{d}u \\ &= \frac{be^{b}}{z} \left( \int_{1}^{\infty} u e^{-bu^2} \, \mathrm{d}u - \int_{0}^{\infty} e^{-bu^2} \, \mathrm{d}u + \int_{0}^{1} e^{-bu^2} \, \mathrm{d}u \right) \\ &= \frac{1}{2z} - \frac{\sqrt{\pi} a e^{b}}{4z^{3/2}} + \frac{be^{b}}{z} \int_{0}^{1} e^{-bu^2} \, \mathrm{d}u. \end{align*}

Then by the principle of analytic continuation, this holds for all $a \in \mathbb{H}_{\text{right}}$ and $z \in \overline{\mathbb{H}_{\text{right}}}$. Then plugging $a = 1-i$ and $z = z_{\pm} = \pm i/\alpha$, we get $b = b_{\pm} = \mp \alpha /2 \in \mathbb{R}$.

\begin{align*} J_{\pm} &= \operatorname{Re}\bigg[ \frac{1}{2z} - \frac{\sqrt{\pi} a e^{b}}{4z^{3/2}} + \frac{be^{b}}{z} \int_{0}^{1} e^{-bu^2} \, \mathrm{d}u \bigg] \\ &= -\frac{\sqrt{\pi} \, e^{b}}{4} \operatorname{Re}\bigg[ \frac{a}{z^{3/2}} \bigg], \end{align*}

By noting that $a_{+}/z_{+}^{3/2} = -\sqrt{2}\,\alpha^{3/2}$ and $a_{-}/z_{-}^{3/2} = i\sqrt{2}\,\alpha^{3/2}$, we get

$$ J_{+} = \frac{\sqrt{2\pi}}{4} \alpha^{3/2} e^{-\alpha/2}, \qquad J_{-} = 0. $$

This complete the proof.

Let $$I(\beta) =\int_{0}^{\infty} x e^{-x} \cos (x) \cos( \beta x^{2}) \, \mathrm dx \, , \quad \beta>0. $$

Let's take the Laplace transform of $I (\beta)$ and then switch the order of integration (which is permissible since the iterated integral converges absolutely).

$$ \begin{align}\mathcal{L} \{I(\beta)\} (s) &= \int_{0}^{\infty} \left(\int_{0}^{\infty} x e^{-x} \cos(x) \cos(\beta x^{2}) \, \mathrm dx \right)e^{-s \beta } \, \mathrm d \beta \\ &= \int_{0}^{\infty} x e^{-x} \cos(x) \int_{0}^{\infty}\cos(\beta x^{2}) e^{- s \beta } \, \mathrm d \beta \, \mathrm dx \\ &= s \int_{0}^{\infty} \frac{x e^{-x}\cos (x)}{x^{4}+s^{2}} \, \mathrm dx \tag{1} \end{align} $$

To evaluate $(1)$, we can integrate the complex function $$f(z) = \frac{z e^{-z}e^{iz}}{z^{4}+s^{2}} $$ around a wedge-shaped contour that makes an angle of $\frac{\pi}{2}$ with the positive real axis (i.e., a closed quarter-circle in the first quadrant of the complex plane).

In the first quadrant of the complex plane, $\vert e^{-z} \vert \le 1$ and $\vert e^{iz} \vert \le 1$. So the integral along the big arc clearly vanishes as the radius of the arc goes to $\infty$.

Integrating around the contour, we get $$ \begin{align} \int_{0}^{\infty} \frac{x e^{-x} e^{ix}}{x^{4}+s^{2}} \, \mathrm dx + \int_{\infty}^{0} \frac{(it)e^{-it} e^{-t}}{t^{4}+s^{2}} \, i \, \mathrm dt &= 2 \pi i \operatorname{Res} \left[f(z), e^{i \pi/4} \sqrt{s} \right] \\ &= 2 \pi i \lim_{z \to e^{i \pi/4} \sqrt{s}}\frac{ze^{-z}e^{iz}}{4z^{3}} \\ &= 2 \pi i \, \frac{e^{i \pi/4}\sqrt{s} \, e^{-\sqrt{2s}} }{4e^{3 \pi i/4}s^{3/2}} \\ &= \pi \, \frac{e^{-\sqrt{2s}}}{2s}. \end{align}$$

And if we equate the real parts on both sides of the equation, we get $$2 \int_{0}^{\infty} \frac{xe^{-x} \cos(x) }{x^{4}+s^{2} } \, \mathrm dx = \pi \, \frac{e^{-\sqrt{2s}}}{2s}.$$

Therefore, $$\mathcal{L} \{I(\beta)\} (s) = \frac{\pi e^{-\sqrt{2s}}}{4}. $$

Due to the uniqueness of the inverse Laplace transform , we only need to show that $$\mathcal{L} \left\{\frac{\sqrt{2 \pi}}{8} \beta^{-3/2} e^{-1/(2 \beta)} \right\} (s) = \frac{\pi e^{-\sqrt{2s}}}{4}$$ in order to prove that $$I(\beta) = \frac{\sqrt{2 \pi}}{8} \beta^{-3/2} e^{-1/(2 \beta)}.$$

From the answers here, we know that for $a,b>0$, $$\int_{0}^{\infty} \frac{\exp \left(-ax^{2}-b/x^{2}\right)}{x^{2}} \, \mathrm dx = - \frac{1}{2} \sqrt{\frac{\pi}{a}} \, \frac{\partial }{\partial b} e^{-2\sqrt{ab}} = \frac{1}{2} \sqrt{\frac{\pi}{b}} \, e^{-2\sqrt{ab}}.$$

(Differentiation under the integral sign is permissible since for any positive $c$ less than $b$, $\frac{\exp \left(-ax^{2}-b/x^{2}\right)}{x^{2}}$ is dominated by the integrable function $ \frac{\exp \left(-ax^{2}-c/x^{2}\right)}{x^{2}}$.)

Therefore, $$\begin{align} \mathcal{L} \left\{\frac{\sqrt{2 \pi}}{8} \beta^{-3/2} e^{-1/(2 \beta)} \right\} (s) &= \frac{\sqrt{2 \pi}}{8} \int_{0}^{\infty}\beta^{-3/2} e^{-1/(2 \beta)} e^{-s \beta } \, \mathrm d \beta \\ &= \frac{\sqrt{2 \pi}}{4} \int_{0}^{\infty} \frac{\exp \left(-s u^{2} - (1/2)/u^{2} \right)}{u^{2}} \mathrm du \\ &= \frac{\sqrt{2 \pi}}{4} \frac{\sqrt{2 \pi}}{2} \, e^{-\sqrt{2s}} \\ &= \frac{ \pi \, e^{-\sqrt{2s}}}{4}. \end{align}$$