Why can $\int_{x=0}^{\infty} x\,\mathrm{e}^{-\alpha x^2}\mathrm {d}x$ not be evaluated by parts to obtain $\frac{1}{2\alpha}$?

One typical way to write integration by parts is to identify your original integral as $\int u \, dv$ and then we can use the identity $$\int u \, dv = uv - \int v \, du$$ Where we form $du$ from $u$ by differentiating $u$ with respect to the independent variable, and we form $v$ from $dv$ by antidifferentiating.

In your case, if I am taking you rightly you have $u = x$ and $dv = e^{-\alpha x^2} dx$. To proceed you need to find $v$, which would be an antiderivative of $e^{-\alpha x^2}$. You won't be able to find a closed form for this, but you can at least write $v(x) = \int_0^x e^{-\alpha t^2} \, dt$, thanks to the Fundamental Theorem of Calculus.

This would change your proposed integration by parts result. You would end up with an antiderivative for your original integrand of: $$x \int_0^x e^{-\alpha t^2} \, dt - \int \int_0^x e^{-\alpha t^2} \,dt\, dx$$

Now, when you are attempting to do a definite integral by parts, I typically prefer to first compute an antiderivative (indefinite integral) by parts and then simply use FTC. So to adapt your example, if I wished to know $\int_0^{\pi} x \sin x \, dx$, I might first use integration by parts as you did to find an antiderivative and then I would have $$\int_0^{\pi} x \sin x \, dx = \left.\sin x - x \cos x\right|_0^{\pi} = 0 - (-\pi) - (0 - 0)$$ But, you can do (proper) definite integrals by parts just by using the same endpoints on the resulting integral, and evaluating the "uv" part across those endpoints. Your integral is improper though, so this introduces an extra step: $$\int_0^{\infty} x e^{-\alpha x^2} \, dx = \lim_{b \to \infty} \int_0^bx e^{-\alpha x^2} \, dx $$ Applying the above and abusing notation, $$\int_0^bx e^{-\alpha x^2} \, dx = \left.\left( x \int e^{-\alpha x^2} \, dx \right)\right|_0^b - \int_0^b \left(\int e^{-\alpha x^2} \, dx \right) \, dx$$

If we go on to use $F(x) = \int_0^x e^{-\alpha t^2} \, dt$ as our antiderivative, this would yield $$\int_0^bx e^{-\alpha x^2} \, dx = \left. x F(x) \right|_0^b - \int_0^b F(x) \, dx$$

But unfortunately this does not lead anywhere helpful. The first term becomes $bF(b) - 0 = bF(b)$. As you have observed, as $b \to \infty$ $F(b) \to \frac{1}{2}\left(\frac{\pi}{\alpha}\right)^{\frac{1}{2}}$, but since there is a factor of $b$ the first term simply goes to $\infty$. The second term I think is a bit difficult to analyze. Maybe there is a clever way to address it but I don't know it.

I think you are going to be unsatisfied with the ultimate answer here because where this is going is this: you found by a simpler method that this integral has a finite value. You used integration by parts (made a mistake) but arrived at an expression of the type $\infty - \infty$. That is actually where this method would lead regardless of the mistake. The final answer is that this is not a contradiction with your original answer. Limits of the type $\infty - \infty$ are indeterminate, which means it could be anything! It could be infinite, or finite. Further analysis is required to find the answer. As it happens, you had already done the further analysis and found the answer was $\frac{1}{2\alpha}$.

Let $u = a \, x^{2}$ then $du = 2 a \, x \, dx$. In the integral presented the differential unit is part of the integrand. With this in mind then \begin{align} I &= \int_{0}^{\infty} x \, e^{-a \, x^{2}} \, dx = \frac{1}{2a} \, \int_{0}^{\infty} e^{- u} \, du = \frac{1}{2a}. \end{align}

Now, if integration by parts is considered then the following is obtained. \begin{align} I &= \left[ \frac{x^{2}}{2} \, e^{-a x^{2}} \right]_{0}^{\infty} + a \, \int_{0}^{\infty} x^{3} \, e^{-a x^{2}} \, dx = a \, \int_{0}^{\infty} x^{3} \, e^{-a x^{2}} \, dx \\ &= \frac{a^{2}}{2} \, \int_{0}^{\infty} x^{5} \, e^{-a x^{2}} \, dx \\ &= \cdots. \end{align} One can define the integral as $I_{1}$ and from integration by parts show that $$I_{1} = \frac{a^{2n-1}}{(2n-1)!} \, I_{2n+1}$$ where $$I_{m} = \int_{0}^{\infty} x^{m} \, e^{-a x^{2}} \, dx.$$

In the general sense integration by parts makes integrals, of the type which can be easily calculated, more complicated, but can often be used to develop a series expansion for approximations. To demonstrate this consider the same integral with a different lower limit. \begin{align} J_{b}(a) &= \int_{b}^{\infty} x \, e^{- a x^{2}} \, dx \\ &= \left[ \frac{x^{2}}{2} \, e^{- a x^{2}} \right]_{b}^{\infty} + a \, \int_{b}^{\infty} x^{3} \, e^{- a x^{2}} \, dx \\ &= - \frac{b^{2}}{2} \, e^{-a b^{2}} - \frac{a \, b^{4}}{4} \, e^{-a b^{2}} - \frac{a^{2} \, b^{6}}{12} \, e^{-a b^{2}} - \cdots. \end{align} This presents the integral as a power series in $b$.

If you want to integrate by parts like (1) suggests, you must differentiate $x$ and integrate $e^{-\alpha x^2}$. The first task is clear, the second is impossible in terms of elementary functions. So what is your mistake? Well, it seems to me that you wrongly believe that a primitive of $e^{-\alpha x^2}$ is $\int_0^\infty e^{-\alpha x^2} \, dx$. This cannot be true, since $\int_0^\infty e^{-\alpha x^2} \, dx$ is a real number, and the derivative of a given real number is always zero!