Why is $\int_0^3 \frac{1}{\sqrt{x-3}}\, \mathrm{d}x$ a complex number?

The area under the curve idea applies to real-valued functions. This function is imaginary for almost all values of the area of integration.

The truth is the concept in your mind (i.e., geometric area and integration) applies to real scalar functions with single variable, while you're not integrating a real function.

In your case, assuming the integrand function ($1/\sqrt{x-3}$) to be a real scalar function, it is not defined over the integration interval $(0,3)$. But, if you assume it to be a complex function like $f(z)=1/\sqrt{z-3}$, then you can integrate it over that interval, and of course, you will get an imaginary result since the function has pure imaginary values throughout the integration interval. Obviously, if you change the integrand function to $g(x)=1/\sqrt{3-x}$, then the result would be a real number equivalent to the geometric area you're looking for.