Why and how are moduli spaces of (semi)stable vector bundles well-behaved?

I believe "nice" here means "is a quasi-projective variety."

As for why, the reason is geometric invariant theory, which is roughly a way of looking at moduli problems, or actions of groups (which is roughly the same thing) and picking out a subset of the quotient (which itself is only nice as a stack) which is a quasi-projective variety. So there's a general definition of semi-stable points for any action of an affine algebraic group acting on a projective variety with choice of equivariant projective embedding (it depends on the choice of embedding) and the quotient of the semi-stable points is always a quasi-projective variety.

From a topological viewpoint, I believe the idea is that one wants to have a Hausdorff quotient space. In other words, consider the space of all holomorphic structures on a fixed (topological) vector bundle on a curve. Holomorphic structures can be viewed as differential operators on sections of the bundle, such that a section is holomorphic if and only if this operator evaluates to zero on the section. (See, for example, sections 5 and 7 of Atiyah and Bott's "The Yang-Mills equations over Riemann surfaces.") This makes the space of holomorphic structures (i.e. the space of bundles with a fixed topological type) into an affine space. The group of complex automorphisms of the bundle acts on this space, and the quotient is the moduli space of holomorphic bundles. If you don't restrict to stable bundles, this quotient space fails to be Hausdorff. Atiyah and Bott reference this to Mumford's 1965 GIT book. Actually, they just say that the moduli space of stable bundles is Hausdorff, due to the fact that the orbits of stable bundles are closed. (Hmmm... that really just says points are closed in the quotient...) I don't know how much of this is spelled out in Mumford; in particular, I don't know whether there's a proof in the literature that the full quotient space fails to be Hausdorff.

Since I can't comment yet, I'll have to recommend this in an answer: The book by Huybrechts and Lehn: "The Geometry of the Moduli Space of Sheaves" might be worth a look. I really liked the parts that I read.