Is there a formula for this integral $I(a,b)=\int_0^1 t^{-3/2}(1-t)^{-1/2}\exp\left(-\frac{a^2}{t}-\frac{b^2}{1-t} \right)dt$
I will derive the following result using the convolution theorem for Laplace Transforms:
$$I(a,b) = \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-1/2} \, e^{-b^2/(1-t)} = \frac{\sqrt{\pi}}{|a|} e^{-(|a|+|b|)^2}$$
I assume $a$ and $b$ are $\gt 0$ for the derivation below, but you will see where the absolute values come from. It is very easy to check the correctness of this result with a few numerical examples in, say, Wolfram Alpha.
To begin, I refer you to the derivation of the following LT relation:
$$\int_0^{\infty} dt \, t^{-3/2} \, e^{-1/(4 t)}\, e^{-s t} = 2 \sqrt{\pi} \, e^{-\sqrt{s}}$$
We may rescale this to get
$$\int_0^{\infty} dt \, t^{-3/2} \, e^{-a^2/t}\, e^{-s t} = \frac{\sqrt{\pi}}{a} \, e^{-2 a\sqrt{s}}$$
Now the convolution theorem states that the convolution of the above
$$\int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-3/2} \, e^{-b^2/(1-t)}$$
is equal to the inverse LT of the product of the individual LTs. This is easily expressed as follows:
$$\frac{1}{i 2 \pi} \int_{c-i \infty}^{c+i \infty} ds \, e^s \frac{\pi}{a b} e^{-2 (a+b) \sqrt{s}}$$
Note that this is being evaluated at $t=1$. And we of course know what the this integral evaluates to:
$$\int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-3/2} \, e^{-b^2/(1-t)} = \sqrt{\pi} \left (\frac{1}{a} + \frac{1}{b} \right ) e^{-(a+b)^2}$$
Of course, this is not the integral sought. But we may derive this integral by differentiating with respect to the parameter $b$:
$$\frac{\partial}{\partial b} \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-1/2} \, e^{-b^2/(1-t)} = -2 b \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-3/2} \, e^{-b^2/(1-t)} $$
which means we need to evaluate the following integral:
$$-2 \sqrt{\pi} \int db \, b\, \left (\frac{1}{a} + \frac{1}{b} \right )\, e^{-(a+b)^2} = -\frac{2 \sqrt{\pi}}{a} \int db\, (a+b) e^{-(a+b)^2} = \frac{\sqrt{\pi}}{a} e^{-(a+b)^2} + C$$
Using the fact that the sought-after integral goes to zero as $b \to \infty$, $C=0$ and we get
$$I(a,b) = \int_0^1 dt \, t^{-3/2} \, e^{-a^2/t} \, (1-t)^{-1/2} \, e^{-b^2/(1-t)} = \frac{\sqrt{\pi}}{a} e^{-(a+b)^2}$$
BONUS
Of course, I could have considered the convolution between two different functions:
$$f(t) = t^{-3/2} e^{-a^2/t}$$
$$g(t) = t^{-1/2} e^{-b^2/t}$$
with corresponding LTs
$$\hat{f}(s) = \frac{\sqrt{\pi}}{a} \, e^{-2 a\sqrt{s}}$$
$$\hat{g}(s) = \sqrt{\frac{\pi}{s}} \, e^{-2 b\sqrt{s}}$$
(I will not derive the latter LT right now.) The convolution is then
$$\frac{\pi}{a} \frac{1}{i 2 \pi} \int_{c-i \infty}^{c+i \infty} ds \, e^s \, s^{-1/2} e^{-2 (a+b) \sqrt{s}} = \frac{\sqrt{\pi}}{a} e^{-(a+b)^2}$$