Understanding integration over Orthogonal Group

Ok so for your first question what I think is meant is the following. A matrix $A \in M_{n \times n} (\mathbb{R})$ has $ n^2$ entries. So one can think of any matrix as a vector in $\mathbb{R}^{n^2}$.

That is to say one can define a bijection $\phi : \{1, \ldots,n\} \times \{1, \ldots, n \} \rightarrow \{1, \ldots, n^2 \}$ which induces the following identification of matrices as vectors.

$$\alpha : M_{n \times n} (\mathbb{R}) \rightarrow \mathbb{R}^{n^2} \text{ where }M \rightarrow (M_{\phi(i,j)})$$

Using that we can therefore integrate over functions of the form

$$ f: M_{n \times n} (\mathbb{R}) \rightarrow \mathbb{R}$$

as

$$\int_{A} f dV = \int_{\alpha(A)}f \circ \alpha^{-1}(x) d\mathcal{L} $$

where in your case $A$ is the orthogonal group.

Basically you are taking the matrices and flattening them until they're just vectors.

Obviously the question now is the wether or not our integral depends on our choice of alpha. Luckily for us it does not and this comes down to the integral being invariant under reordering of the coordinates.

what it means by integrating over the orthogonal group

The orthogonal group is a locally compact group (in fact it is compact), therefore it admits a Haar measure (which is both right and left invariant). Basically this is a measure $dV$ on the Borel subsets of $O(n)$ which is invariant under translation by elements in $O(n)$. This should explain the first equality.

Each step of $\int_{O_n} \sum_i M_{ii}dV = n\int_{O_n}M_{11}dV = 0$

Well first we clearly have that $\int_{O_n} \sum_i M_ii dV = \sum_i \int_{O_n} M_{ii} dV$. Consider one of these summands

$$\int_{O_n} M_{ii} dV$$ The point now is that we integrate over all matrices, this term is intuitively zero because for every matrix $M$ there will be a matrix $N$ such that $N_{ii}=-M_{ii}$. Formally we can permute rows and columns of $M$ by multiplying by some elementary matrix so by replacing the $i$'th row with the first row and the $i$'th column with the first column this equals to $\int_{O_n} M_{11} dV$. This proves the first equality.

also by multiplying by the matrix which corresponds to multiplying the first row by $-1$ we have that $$\int_{O_n} M_{11} dV=\int_{O_n} -M_{11} dV$$ hence both the arguments are zero.