Generalisations of Riemann-Roch for surfaces

If $X$ is proper with rational singularities (and quotient and A-D-E (=Du Val) singularities are rational), then you can do most cohomology computations on a resolution.

Let $\pi:Y\to X$ be a resolution of singularities (not necessarily minimal). Then if $X$ has rational singularities, then $R^i\pi_*\mathscr O_Y=0$ for $i>0$. Let $L$ be a line bundle on $X$. Then by the above vanishing, $h^i(Y,\pi^*L)=h^i(X,L)$, so we have that $$\chi(Y,\pi^*L)=\chi(X,L).$$ Note that this actually holds in any dimension.

It follows that one has a sort-of-RR on $X$ (surface): $$\chi(X,L)=\chi(\mathcal{O}_X)+\frac{1}{2}((\pi^*c_1(L))^2-(\pi^*c_1(L))\cdot K_Y).$$ (Remark: it is not a bad idea to distinguish when we talk about a line bundle and when we talk about the associated Cartier divisor! Besides having the formulas be well-defined one has to remember that for example the push forward of a divisor via a birational map is also a divisor, while the push forward of a line bundle is not necessarily a line bundle. And even if it is, the divisor associated to the push forward of a line bundle is not necessarily the same as the push forward of the divisor associated to the line bundle.)

As for your question about the quasi-projective case, Christian already said that it is tricky. In general, when it comes to cohomology, experience shows that it is better to compactify and figure out the difference than trying to develop a handicapped theory for quasi-projective varieties directly. (See Deligne's way of doing Hodge theory on open varieties).

The genus formula on a surface is essentially equivalent to Riemann-Roch, so as soon as the formula makes sense, it will follow.

The adjunction formula in the form $$ \omega_C \cong \omega_X(C)|_C $$ holds whenever $C$ is a Cartier divisor on a Gorenstein scheme $X$. Taking Euler characteristics, you get an extremely general genus formula.

If I remember correctly, you'll find in Beauville's book on algebraic surfaces a remark on what assumptions are needed to set up an intersection theory on algebraic surfaces: you should always get a reasonable theory if $X$ is Cohen-Macaulay, and if you intersect Cartier divisors. Mumford even constructed an intersection theory on normal surfaces.

In case $X$ is not proper, the cohomology groups $h^i({\cal O}_X)$ may not be finite dimensional (e.g., if $X$ is the affine plane with origin removed), and to talk about $\chi$ is thus a little bit tricky - in this case, you may want to look up literature on $L^2$-cohomology.

Finally, if $X$ is proper with quotient singularities, I think you'll find a Riemann-Roch theorem in Miles Reid's "Young person's guide to canonical singularities". But note that even in order to define $K_X^2$ you need $K_X$ to be at least ${\mathbb{Q}}$-Cartier or so (which is OK for quotient singularities), in case $X$ is not Gorenstein, the self-intersection might be a rational number.