Invariants of higher genus curves

For $g > 2$ you're not going to be able to write such invariants as global functions on the moduli space.

In general for an algebraic variety $X$, there is a canonical morphism $a \colon X \to \mathrm{Spec}(H^0(X,\mathcal O_X))$, the affinization of $X$. Points $x$ and $y$ of $X$ are separated by some global regular function if only if $a(x) \neq a(y)$. So your question can only have a positive answer if the affinization morphism is injective (so $M_g$ is affine or quasi-affine). Now $M_2$ is affine, but $M_g$ for $g >2$ contains a complete curve. Since a regular function restricted to a complete subvariety is constant, this means that $a$ has positive dimensional fibers. In fact one can prove using the Satake compactification that through any two points of $M_g$ passes a complete curve, if $g>2$. So the affinization is in fact a point.

This is also why in the $g=3$ case of David Lehavi's answer there are only such choices of invariants on the hyperelliptic locus and on the non-hyperelliptic locus separately. Both loci are affine, but their union is not.

It also seems relevant to mention that $M_g$ is of general type for $g \gg 0$, so that there will be no rational parametrization of the set of genus $g$ curves. And it also seems relevant to mention Fenchel--Nielsen coordinates: these will give you local real-analytic coordinates around any point of the moduli stack in any genus.

These invariants are traditionally called moduli. Riemann found that there are $3g-3$ complex parameters defining a curve of genus $g>1$. The modern statement is that the space of curves of genus $g$ is $3g-3$-dimensional. There are several different ways to introduce coordinates on this moduli space. A common way to look at it is as a factor of Teichmuller space over the action of the Teichmuller modular group. For genus 1 (elliptic curves), Teichmuller space is the upper half-plane, the modular group is the usual modular group. The $j$-invariant maps the upper half-plane of this factor.

In the general case, Teichmuller space is biholomorphic to a bounded region in $C^{3g-3}$.

The literature on the topic is enormous. The key words are: Teichmuller space, and Moduli spaces of curves.

I recommend to begin with Ahlfors, Lectures on quasiconformal mappings, and Abikoff, Real-analytic theory of Teichmuller spaces.

AFAIK, these are know only up to genus 3.

• Genus 2: Igusa (classical).
• Hyperelliptic genus 3: Shioda (classical).
• Non hyperelliptic genus 3: a decade ago by Dixmier & Ohno - see https://www.win.tue.nl/~aeb/math/ternary_quartic.html