Integral of $x^2 \exp(-ax^2)$ using the gamma function?

Summarising the comments, after substitution of $u=ax^2$ and $du=2axdx$, you'll get $$ \int_0^\infty x^2 e^{-ax^2}dx=\frac{1}{2a^{3/2}}\int_0^\infty u^{3/2-1}e^{-u} du=\frac{\Gamma(3/2)}{2a^{3/2}}=\frac{\sqrt{\pi}}{4a^{3/2}}. $$

You can write the integral in this form: $$d/da \int{\exp{(-ax^2)}dx}$$ And we know that $$\int{\exp{(-ax^2)}dx}=\sqrt{\pi/a}$$ So $$\int{-x^2\exp{(-ax^2)}dx}=d/da (\sqrt{\pi/a})=\sqrt{\pi}/2a^{3/2}$$