When are GIT quotients projective?
I'm not sure if this is the sort of thing you are after but one can say the following.
Suppose we work over a base field $k$. If $X$ is proper over $k$ and the $G$-linearized invertible sheaf $L$ is ample on $X$ then the uniform categorical quotient of $X^{ss}(L)$ by $G$ is projective and so it gives a natural compactification of $X^s(L)/G$.
I believe this is mentioned in Mumford's book but no proof is given. It goes via considering $\operatorname{Proj}R_0$ where $R_0$ is the subring of invariant sections of $$\bigoplus_i H^0(X,L^{\otimes i})$$ and showing that this is the quotient.
This is stated in Theorem 1.12 in these notes of Newstead. I haven't found a proof written down - but either showing it directly or doing it by reducing to the case of projective space shouldn't be too hard I don't think.
If your desired moduli space can be interpreted as a moduli space of quiver representations, then you may be in luck. For moduli of $\theta$-(semi)stable quiver representations, there is a simple criterion equivalent to the stable and semistable loci being equal: the dimension vector $\alpha$ is indecomposable (i.e., not a non-trivial sum of elements) in $\theta^\perp\cap \mathbb Z^{Q_0}_{\geq 0}$, where $Q_0$ is your set of vertices, and $\theta$ is the fixed stability parameter.
To learn more about moduli of quiver representations, I recommend King's paper "Moduli of representations of finite dimensional algebras" (MR).
Is the answer "when there are no non-constant invariant functions" too stupid? (By construction, functions on the GIT quotient are invariant global functions on $X$). Because that seems to show that $X$ projective implies $X^{ss}(L)/G$ projective.