Zariski tangent spaces to representation varieties

So I don't think I am going to answer your question here, and I think you probably know most of what I am going to write at this point. But perhaps this information will be of use to others that happen to pass by this thread.

Let $G$ be algebraic (including compact, and complex reductive) and denote $R(G)=\mathrm{Hom}(\pi,G)$.

The tangent space of $R(G)$ at $\rho$ always make sense since it is algebraic and it is the twisted cocycles $Z^1(\pi,\mathfrak{g}_{\mathrm{Ad}\rho})$.

Likewise, the tangent space to a conjugation orbit also generally makes sense and is the twisted coboundaries $B^1(\pi,\mathfrak{g}_{\mathrm{Ad}\rho})$.

Both of these are results of Weil from Remarks on cohomology of groups, I believe.

Unfortunately the tangent space to a quotient is not always the quotient of the tangent spaces, so it is not true that the tangent space to $R(G)/G$ is always $H^1(\pi,\mathfrak{g}_{\mathrm{Ad}\rho})$. Note that if $G$ is compact this quotient is the orbit space, and if $G$ is complex reductive it is the GIT quotient.

An easy example is the following: Let $\pi=\mathbb{Z}$ and let $G=\mathrm{SL}(n,\mathbb{C})$. Then the GIT quotient $R(G)/G$ is $\mathbb{C}^{n-1}$, parameterized by coefficients of the characteristic polynomial, and so is smooth. Thus the tangent space to the identity character is $\mathbb{C}^{n-1}$. On the other hand, the orbit of the identity is trivial and it is a smooth point in $R(G)=G$. So the cocycles are $\mathfrak{g}=\mathbb{C}^{n^2-1}$ and the coboundaries are trivial. Thus the first cohomology is also $\mathbb{C}^{n^2-1}$ which is much bigger than the correct result of $\mathbb{C}^{n-1}$.

However the following is true. If $\rho \in R(G)$ is smooth (always true for irreducible reps in surface groups, and all reps in twisted surface groups and all reps in free groups), and also have a closed conjugation orbit, then using an appropriate Slice Theorem (Luna, or Mostow) one can prove that the tangent space at the equivalence class $[\rho]$ is the tangent space at 0 in a quotient of cohomology: $T_0(H^1(\pi,\mathfrak{g}_{\mathrm{Ad}\rho})/C)$, where $C$ is the centralizer of $\rho$ in $G$. But in general there are known counter examples. See here.

Returning to the above example, we then have $\mathfrak{sl}(n,\mathbb{C})/C=\mathbb{C}^{n-1}$ since $C=\mathrm{SL}(n,\mathbb{C})$ (still parameterized by the coefficients of the characteristic polynomial, but now instead of the determinant being fixed, the trace is fixed).

If the centralizer is acts trivially one recovers the usual statement that the tangent space is $H^1(\pi,\mathfrak{g}_{\mathrm{Ad}\rho})$. For example, this holds for irreducible representations and $G=\mathrm{GL}(n,\mathbb{C})$ or $\mathrm{SL}(n,\mathbb{C})$ (or their maximal compact subgroups), and $\pi$ is a free group or a (twisted) surface group. But even when $\pi$ is a free group and $G=\mathrm{PSL}(2,\mathbb{C})$ and $\rho$ is an irreducible representation there are counter examples (namely there are irreducible singularities in that case).

There is always an open dense subset of smooth irreducibles and a further subset of those whose centralizer is equal to the center of $G$. These are called "good" representations (a la Millson). The set of good representations modulo G is a manifold, and the set of all irreducibles forms an orbifold (for general $G$ irreducible means that $\rho(\pi)$ is not contained in a parabolic subgroup).

Let me try to briefly address (1). Whenever the dimension of the tangent space is the dimension of the space, in the strong topology there is a neighborhood that looks like a ball. The converse is not true in general.

Think of $x^2=y^3$ which has a singularity at $(0,0)$ algebraically but the neighborhood of $(0,0)$ in the variety is homeomorphic to a Euclicean neighborhood. On the other hand $xy=0$ also has a dimension jump in tangent space at $(0,0)$ but no neighborhood is homeomorphic to a ball. Anyway, singularities in an algebraic setting can be mild or wild or something in between (like orbifold type).

With respect to (1), if $[\rho]$ is in the interior of a maximal simplex, I think it might still depend on $[\rho]$. Anyway, in general (as the example $x^2=y^3$ shows) knowing that a point in a semi-algebraic set has a neighborhood homeomorphic to a ball does not tell you it is not singular.

Note: A recent paper of Millson and Kapovich shows that singularities in character varieties can get as bad as you can imagine. See here.

For $\pi$ a free group however, I believe generally (for all but a finite number of counter examples) the situation is this: reducibles are singular and have no neighborhood homeomorphic to a ball. For $\pi$ a free group and $G$ equal to $\mathrm{GL}(n,\mathbb{C})$ or $\mathrm{SL}(n,\mathbb{C})$ this more or less has been established. See my paper here. Also the irreducibles are either smooth or admit orbifold singularieties. But there are orbifolds that are homeomorphic to manifolds (but not always), so it seems that one could have an orbifold singularity (with an irreducible) that happens to have a neighborhood homeomorphic to a ball (this would give a negative answer to (1)). I don't however have an example to show this for certain.

About (2), if the cohomology corresponds to the tangent space (like for free groups, or twisted surface groups), then yes, at a singularity, the dimension must jump, and hence the cohomology does too. But if the point does not correspond to cohomology to begin with, then although the tangent space must still have a dimension jump, the cohomology conceivably could not. Again, I don't have an example to show this for certain.

It is false in general. A counter-example was constructed by J.Huebschmann in section 6 of his paper "Singularities and Poisson geometry of certain representation spaces". However, if, say, $\rho(\Gamma)$ has trivial centralizer, then the Zariski tangent space to $Hom(\Gamma, G)//G$ (you need to use Mumford quotient for the question to make sense) at $[\rho]$ is indeed isomorphic to $H^1(\Gamma, Ad \rho)$. The reason is that the $G$-action at such $\rho$'s admits a local cross-section and then everything works, as it was explained by A.Weil in his 1964 paper "Remarks on cohomology of groups".

There are some people around who know much more about this subject than I do, but since none of them has responded so far, let me make a few remarks. By an old theorem of Narasimhan and Seshadri (1965), moduli of flat unitary bundles with irreducible holonomy and of holomorphic stable bundles on a Riemann surface $\Sigma$ are isomorphic. The former space is basically the representation variety you've described and has a canonical symplectic structure; the latter space is a Mumford GIT quotient, so it's an algebraic variety. There is a big machine which implies such a relation between the Kahler/symplectic reduction and a GIT quotient for finite-dimensional group actions, except that the group in this situation is the gauge group, so some care needs to be taken.

Irreducibility of the representation $\rho$ assures that the holonomy of the corresponding connection has trivial centralizer (scalar matrices) and that the corresponding point $[\rho]$ in the moduli space is regular (non-singular and non-orbifold). Without it, you have to be careful about defining the quotient. The first cohomology group

$$\text{H}^1(\Sigma, \operatorname{End}\rho)=\text{H}^1(\pi, \operatorname{End}\rho)$$ classifies $G$-conjugacy classes of infinitesimal deformations of $\rho:\pi\to G$ by abstract Kodaira-Spencer theory, so the answer to Q1 is affirmative: if the point $[\rho]$ is smooth, the dimension of the moduli space at $[\rho]$ is the dimension of the first cohomology. This statement generalizes to arbitrary finitely generated discrete groups $\Gamma$. You can let $\Sigma=\text{K}(\pi,1)$ and consider topological cohomology of $\Sigma$ instead of group cohomology of $\pi$. The case studied by Weil is when $\Sigma$ is a hyperbolic $n$-manifold, so that the moduli space no longer has complex or algebraic structure for $n\geq 3.$ One difficulty in addressing Q2 is that if $\rho$ is reducible, you get an orbifold point, so it doesn't quite make sense to speak about triangulations. Another difficulty is that if the cohomological dimension of $\pi$ is greater than 2 (i.e. $\dim \Sigma\geq 3$), it becomes harder to keep track of the dimensions of higher cohomology groups by means of the Euler characteristic. Representation varieties of 3-manifold groups have been extensively studied (e.g. look at papers of Shalen).