Closed-form expression for $\int_{0}^{1}e^{-ax(1 - bx )}x^{\alpha-1}(1-x)^{\beta - 1}dx$?
The problematic component of the proposed integral is the term $e^{ab x^2}$. This can be eliminated by expanding this exponential into the corresponding power series for which the integral in question becomes \begin{align} I &= \sum_{n=0}^{\infty} \frac{(ab)^{n}}{n!} \, \int_{0}^{1} \, e^{-ax} \, x^{2n + \mu-1} \, (1-x)^{\nu - 1} \, dx \\ &= \sum_{n=0}^{\infty} \frac{(ab)^{n}}{n!} \, B(2n+\mu, \nu) \, {}_{1}F_{1}(2n+\mu; 2n+\mu+\nu; -a) \\ &= B(\mu, \nu) \, \sum_{n=0}^{\infty} \sum_{r=0}^{\infty} \frac{(\mu)_{2n} (2n+\mu)_{r} }{ (\mu + \nu)_{2n} (2n+\mu + \nu)_{r} } \, \frac{(ab)^{n}}{n!} \, \frac{(-a)^{r}}{r!} \\ &= B(\mu, \nu) \, \sum_{n=0}^{\infty} \sum_{r=0}^{\infty} \frac{(\mu)_{2n+r}} { (\mu + \nu)_{2n+r} } \, \frac{(ab)^{n}}{n!} \, \frac{(-a)^{r}}{r!} \\ &= B(\mu, \nu) \, F_{1:0}^{1:0} \left[ \begin{array}{cc} [(\mu):2,1]: - ; \\ [(\mu + \nu):2,1]: - ; \end{array} \hspace{3mm} ab , \, -a \right] \end{align} where the last function, $F$, is the Srivastava-Daoust function. From this it can be stated that \begin{align} \frac{1}{B(\mu, \nu)} \, \int_{0}^{1} \, e^{-ax(1-bx)} \, x^{\mu-1} \, (1-x)^{\nu - 1} \, dx = F_{1:0}^{1:0} \left[ \begin{array}{cc} [(\mu):2,1]: - ; \\ [(\mu + \nu):2,1]: - ; \end{array} \hspace{3mm} ab , \, -a \right] \end{align}
The Srivastava-Daoust Function is defined by formula (1.1) in the article by Rekha Panda
$\dfrac{1}{B(\alpha,\beta)}\int_0^1e^{-ax(1-bx)}x^{\alpha-1}(1-x)^{\beta-1}~dx$
$=\dfrac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\int_0^1x^{\alpha-1}(1-x)^{\beta-1}e^{-ax+abx^2}~dx$
$=\dfrac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\int_0^1\sum\limits_{n=0}^\infty\dfrac{a^nb^nx^{2n+\alpha-1}(1-x)^{\beta-1}e^{-ax}}{n!}~dx$
$=\dfrac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\sum\limits_{n=0}^\infty \dfrac{a^nb^nB(2n+\alpha,\beta){_1}F_1(2n+\alpha;2n+\alpha+\beta;-a)}{n!}$ (according to https://en.wikipedia.org/wiki/Confluent_hypergeometric_function#Integral_representations)
$=\dfrac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{(-1)^k\Gamma(2n+\alpha)\Gamma(\beta)(2n+\alpha)_ka^{n+k}b^n}{\Gamma(2n+\alpha+\beta)n!(2n+\alpha+\beta)_kk!}$
$=\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{(-1)^k(\alpha)_{2n}\Gamma(2n+k+\alpha)\Gamma(2n+\alpha+\beta)a^{n+k}b^n}{(\alpha+\beta)_{2n}\Gamma(2n+\alpha)\Gamma(2n+k+\alpha+\beta)n!k!}$
$=\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{(-1)^{2k}\Gamma(\alpha+\beta)\Gamma(2n+2k+\alpha)a^{n+2k}b^n}{\Gamma(\alpha)\Gamma(2n+2k+\alpha+\beta)n!(2k)!}+\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{(-1)^{2k+1}\Gamma(\alpha+\beta)\Gamma(2n+2k+\alpha+1)a^{n+2k+1}b^n}{\Gamma(\alpha)\Gamma(2n+2k+\alpha+\beta+1)n!(2k+1)!}$
$=\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{(\alpha)_{2(n+k)}a^{n+2k}b^n}{(\alpha+\beta)_{2(n+k)}4^kn!k!\left(\dfrac{1}{2}\right)_k}-\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{(\alpha)_{2(n+k)+1}a^{n+2k+1}b^n}{(\alpha+\beta)_{2(n+k)+1}4^kn!k!\left(\dfrac{3}{2}\right)_k}$ (according to http://mathworld.wolfram.com/PochhammerSymbol.html)
$=\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{\left(\dfrac{\alpha}{2}\right)_{n+k}\left(\dfrac{\alpha+1}{2}\right)_{n+k}a^{n+2k}b^n}{\left(\dfrac{\alpha+\beta}{2}\right)_{n+k}\left(\dfrac{\alpha+\beta+1}{2}\right)_{n+k}4^kn!k!\left(\dfrac{1}{2}\right)_k}-\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{\left(\dfrac{\alpha}{2}\right)_{n+k+1}\left(\dfrac{\alpha+1}{2}\right)_{n+k}a^{n+2k+1}b^n}{\left(\dfrac{\alpha+\beta}{2}\right)_{n+k+1}\left(\dfrac{\alpha+\beta+1}{2}\right)_{n+k}4^kn!k!\left(\dfrac{3}{2}\right)_k}$
(according to http://mathworld.wolfram.com/PochhammerSymbol.html)
$=\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{\left(\dfrac{\alpha}{2}\right)_{n+k}\left(\dfrac{\alpha+1}{2}\right)_{n+k}a^{n+2k}b^n}{\left(\dfrac{\alpha+\beta}{2}\right)_{n+k}\left(\dfrac{\alpha+\beta+1}{2}\right)_{n+k}4^kn!k!\left(\dfrac{1}{2}\right)_k}-\sum\limits_{n=0}^\infty \sum\limits_{k=0}^\infty\dfrac{\alpha\left(\dfrac{\alpha+1}{2}\right)_{n+k}\left(\dfrac{\alpha+2}{2}\right)_{n+k}a^{n+2k+1}b^n}{(\alpha+\beta)\left(\dfrac{\alpha+\beta+1}{2}\right)_{n+k}\left(\dfrac{\alpha+\beta+2}{2}\right)_{n+k}4^kn!k!\left(\dfrac{3}{2}\right)_k}$
In fact it is not indeed to express in terms of Srivastava-Daoust Function, it can just express in terms of Kampé de Fériet function.