Gaussian integral with imaginary coefficients and Wick rotation
Proposition. Let there be given two complex numbers $a,b\in \mathbb{C}$ such that ${\rm Re}(a)\geq 0$. In the case ${\rm Re}(a)=0$, we demand furthermore that ${\rm Im}(a)\neq 0$ and ${\rm Re}(b)=0$. The Gaussian integral is well-defined and is given by $$ \underbrace{\int_{\mathbb{R}}\!dx~ e^{-\frac{a}{2}x^2+bx}}_{=:~ I_{\mathbb{R}}(a,b)} ~=~\lim_{\begin{array}{c} x_i\to -\infty \cr x_f\to \infty \end{array} } \underbrace{\int_{[x_i,x_f]}\!dx~ e^{-\frac{a}{2}x^2+bx}}_{=:~ I_{[x_i,x_f]}(a,b)} ~=~\underbrace{\sqrt{\frac{2\pi}{a}}e^{\frac{b^2}{2a}}}_{=:~ F(a,b)}, \tag{A}$$ where it is implicitly understood that the square root has positive real part.
Remark: The Riemann/Darboux integral is not defined for non-bounded sets, so it can only be used for the middle expression of eq. (A).
I) Sketched proof in case of ${\rm Re}(a)> 0$: The function $g(x)=e^{-\frac{{\rm Re}(a)}{2}x^2+{\rm Re}(b)x}$ serves as a majorant function for Lebesgue's dominated convergence theorem, which establishes the first equality of eq. (A). For the second equality of eq. (A), we divide the proof into cases:
Case $a>0$ and $b\in \mathbb{R}$. Complete the square. $\Box$
Case $a>0$. Complete the square. Shift the integration contour appropriately to a horizontal line in the complex plane in order to reduce to case 1, cf. Cauchy's integral theorem. Argue that contributions at infinity vanish. $\Box$
Case ${\rm Re}(a)> 0$. Rotate the integration contour to a line of steepest descent in order to reduce to case 2, cf. Cauchy's integral theorem. Argue that contributions at infinity vanish. $\Box$
II) Sketched proof in the oscillatory case ${\rm Re}(a)=0, {\rm Im}(a)\neq 0, {\rm Re}(b)=0$: The lhs. of eq. (A) is not Lebesgue integrable. It is an improper integral defined via the middle expression of eq. (A). It remains to prove the second equality of eq. (A). It is possible to give a proof using Cauchy's integral theorem along the lines of Jack's answer. In this answer we will instead give a proof in the spirit of an infinitesimal deformation prescription.
Given $\varepsilon>0$. As $x_i\to \infty$ and $x_f\to \infty$ it is not hard to see that $I_{[x_i,x_f]}(a,b)$ oscillates with smaller and smaller amplitude that tends to zero, and it is hence convergent without any regularization. The convergence improves if we let $a$ have a positive real part. In other words, the convergence is uniform wrt. ${\rm Re}(a)\geq 0$, i.e.
$$ \exists X_i,X_f\in \mathbb{R} ~\forall x_i\leq X_i~\forall x_f\geq X_f ~\forall {\rm Re}(a)\geq 0:~~ \left| I_{[x_i,x_f]}(a,b)- I_{\mathbb{R}}(a,b)\right| ~\leq~\frac{\varepsilon}{4}.\tag{B}$$
Next use Lebesgue's dominated convergence theorem with majorant function of the form $g(x)=C~1_{[x_i,x_f]}(x)$ (where $C>0$ is an appropriate constant) to argue that
$$I_{[x_i,x_f]}( i{\rm Im}(a),b) ~=~\lim_{{\rm Re}(a)\to 0^+} I_{[x_i,x_f]}(a,b) , \tag{C}$$
i.e. $\exists {\rm Re}(a)>0$ such that
$$ \left| I_{[x_i,x_f]}( i{\rm Im}(a),b)-I_{[x_i,x_f]}( a ,b) \right| ~\leq~\frac{\varepsilon}{4},\tag{D}$$
and
$$\left| \underbrace{F( a ,b)}_{=~ I_{\mathbb{R}}(a,b)}- F( i{\rm Im}(a),b) \right| ~\leq~\frac{\varepsilon}{4}.\tag{E}$$
In eq. (E) we used that the function $F$ is continuous. All together, eqs. (B), (D) & (E) yield $$ \begin{align} \left| I_{\mathbb{R}}(i{\rm Im}(a),b) - F( i{\rm Im}(a),b)\right| ~\leq~&\left| I_{\mathbb{R}}(i{\rm Im}(a),b)- I_{[x_i,x_f]}( i{\rm Im}(a),b)\right|\cr &+\left| I_{[x_i,x_f]}( i{\rm Im}(a),b) - I_{[x_i,x_f]}( a ,b)\right| \cr &+\left| I_{[x_i,x_f]}(a,b)- I_{\mathbb{R}}(a,b)\right|\cr &+\left|F( a ,b) - F( i{\rm Im}(a),b)\right| \cr ~\leq~&\varepsilon.\end{align} \tag{F}$$
Eq. (F) shows that the second equality of eq. (A) holds. $\Box$
First of all, one need to prove the following key integral formula: \begin{align*} \boxed{I = \int_{-\infty}^{+\infty} d x e^{ i a x^2} = \sqrt{\dfrac{i \pi}{a}} \qquad (a>0) } \end{align*} Usually one can pick up an analytic function $f(z)=e^{ i a z^2}$ and then performs the complex integral along the closed contour showed above. $$J = \oint d z e^{ i a z^2} = \int_{C_1} + \int_{C_2} + \int_{C_3} + \int_{C_4} = J_1 + J_2 + J_3 + J_4 =0$$ Next one need calculate the four line integrals along four different intervals.
- $C_1:z=x \qquad x \in [-p,p]$ $$J_1 = \int_{C_1} dz e^{i a z^2} = \int_{-p}^p d x e^{i a x^2}$$
- $C_2:z=p+iy \qquad y \in [0,p]$ $$J_2 = \int_{C_2} dz e^{i a z^2} = \int_0^p i d y e^{i a (p+i y)^2}$$
- $C_3:z=x+iy = (1+i) y = \sqrt{2}e^{i\pi/4}y \qquad y \in [p,-p]$ \begin{align*} J_3 & = \int_{C_3} dz e^{i a z^2} = e^{i\pi/4} \int_{\sqrt{2}p}^{-\sqrt{2}p} d x e^{- a x^2} \end{align*}
- $C_4:z=-p + i y \qquad y \in [-p,0]$ \begin{align*} J_4 & = \int_{C_4} dz e^{i a z^2} = \int_{-p}^0 i d y e^{i a (i y -p)^2} \qquad (y \Rightarrow -y) \\ & = \int_p^0 - i d y e^{i a (-i y - p)^2} = \int_0^p i d y e^{i a (i y+p)^2} \\ & = J_2 \\ \end{align*}
In the final step, we will consider the limiting process:$p\rightarrow+\infty$ \begin{align*} & 0 \leq |J_2| \leq \left|\int_0^p i d y e^{i a (i y+p)^2}\right| \leq \int_0^p d y e^{-2 a p y} = \dfrac{1-e^{-2a p^2}}{2 a p} \\ & \lim_{p \rightarrow +\infty} \dfrac{1-e^{-2a p^2}}{2 a p} \Rightarrow \lim_{p\rightarrow+\infty} |J_2| = 0 \Rightarrow \boxed{\lim_{p\rightarrow +\infty} J_2 = \lim_{p\rightarrow +\infty} J_4 =0} \end{align*} \begin{align*} & p\rightarrow +\infty \Rightarrow J_1+J_3=0 \Rightarrow J_1 = -J_3 = e^{i\pi/4} \int_{-\infty}^{+\infty} d x e^{- a x^2} = e^{i \pi/4} \sqrt{\dfrac{\pi}{a}} = \sqrt{\dfrac{i\pi}{a}} \\ & \Rightarrow \boxed{\int_{-\infty}^{+\infty} d x e^{i a x^2} = \sqrt{\dfrac{i\pi}{a}} \qquad (a>0)} \end{align*} One can also derive a similar result for $a<0$ with the help of the following closed contour.
Then for the integral you are giving: \begin{align} \int_{-\infty}^{+\infty} e^{\dfrac{iax^2}{2}+iJx} dx = \int_{-\infty}^{+\infty} e^{i \dfrac{a}{2}(x+\dfrac{J}{a})^2-i\dfrac{J^2}{2a}} dx = \sqrt{\dfrac{i2\pi}{a}}e^{-i\dfrac{J^2}{2a}} \end{align}
Hope it helps.