Is there a formula for this integral $I(a,b)=\int_0^1 t^{-3/2}(1-t)^{-1/2}\exp\left(-\frac{a^2}{t}-\frac{b^2}{1-t} \right)dt$

I will derive the following result using the convolution theorem for Laplace Transforms:

$$I(a,b) = \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-1/2} \, e^{-b^2/(1-t)} = \frac{\sqrt{\pi}}{|a|} e^{-(|a|+|b|)^2}$$

I assume $a$ and $b$ are $\gt 0$ for the derivation below, but you will see where the absolute values come from. It is very easy to check the correctness of this result with a few numerical examples in, say, Wolfram Alpha.

To begin, I refer you to the derivation of the following LT relation:

$$\int_0^{\infty} dt \, t^{-3/2} \, e^{-1/(4 t)}\, e^{-s t} = 2 \sqrt{\pi} \, e^{-\sqrt{s}}$$

We may rescale this to get

$$\int_0^{\infty} dt \, t^{-3/2} \, e^{-a^2/t}\, e^{-s t} = \frac{\sqrt{\pi}}{a} \, e^{-2 a\sqrt{s}}$$

Now the convolution theorem states that the convolution of the above

$$\int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-3/2} \, e^{-b^2/(1-t)}$$

is equal to the inverse LT of the product of the individual LTs. This is easily expressed as follows:

$$\frac{1}{i 2 \pi} \int_{c-i \infty}^{c+i \infty} ds \, e^s \frac{\pi}{a b} e^{-2 (a+b) \sqrt{s}}$$

Note that this is being evaluated at $t=1$. And we of course know what the this integral evaluates to:

$$\int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-3/2} \, e^{-b^2/(1-t)} = \sqrt{\pi} \left (\frac{1}{a} + \frac{1}{b} \right ) e^{-(a+b)^2}$$

Of course, this is not the integral sought. But we may derive this integral by differentiating with respect to the parameter $b$:

$$\frac{\partial}{\partial b} \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-1/2} \, e^{-b^2/(1-t)} = -2 b \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-3/2} \, e^{-b^2/(1-t)} $$

which means we need to evaluate the following integral:

$$-2 \sqrt{\pi} \int db \, b\, \left (\frac{1}{a} + \frac{1}{b} \right )\, e^{-(a+b)^2} = -\frac{2 \sqrt{\pi}}{a} \int db\, (a+b) e^{-(a+b)^2} = \frac{\sqrt{\pi}}{a} e^{-(a+b)^2} + C$$

Using the fact that the sought-after integral goes to zero as $b \to \infty$, $C=0$ and we get

$$I(a,b) = \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-1/2} \, e^{-b^2/(1-t)} = \frac{\sqrt{\pi}}{a} e^{-(a+b)^2}$$

BONUS

Of course, I could have considered the convolution between two different functions:

$$f(t) = t^{-3/2} e^{-a^2/t}$$

$$g(t) = t^{-1/2} e^{-b^2/t}$$

with corresponding LTs

$$\hat{f}(s) = \frac{\sqrt{\pi}}{a} \, e^{-2 a\sqrt{s}}$$

$$\hat{g}(s) = \sqrt{\frac{\pi}{s}} \, e^{-2 b\sqrt{s}}$$

(I will not derive the latter LT right now.) The convolution is then

$$\frac{\pi}{a} \frac{1}{i 2 \pi} \int_{c-i \infty}^{c+i \infty} ds \, e^s \, s^{-1/2} e^{-2 (a+b) \sqrt{s}} = \frac{\sqrt{\pi}}{a} e^{-(a+b)^2}$$