Prove that $\det( M) \in \mathbb{Z}$.
Let $G_1,G_2, \dots, G_k$ be the elements of $\Gamma$.
$$M^2 = (G_1+G_2+ \dots+ G_k)^2 = \sum_{i=1}^{k}G_i\left(\sum_{j=1}^k G_j\right)$$ $$= \sum_{i=1}^{k}G_i \left(\sum_{G \in \Gamma} G_i^{-1}G\right) = \sum_{G \in \Gamma} \sum_{i=1}^{k}G_i(G_i^{-1}G) = k\sum_{G \in \Gamma}G = kM$$ Taking determinants, we get that $(\det M)^2 = k^n \det M$. Hence either $\det M = 0$ or $\det M$ is equal to the order of $\Gamma$ raised to the $n$th power.
This can also be proved using basic representation theory of finite groups (this may be an overkill, though).
A group $\Gamma$ of $N\times N$ matrices can be seen as an abstract group equiped with a representation on a N-dimensional complex vector space. $\frac{M}{|\Gamma|}$ is well-known to be the projection onto the subspace of invariants. Now, concider two cases:
- The whole vector space is invariant. Thus, the subspace of invariants coincides with the whole space, and the projection operator coincides with the identity: $\frac{M}{|\Gamma|} = 1$. Taking determinants, we have $\frac{\det M}{|\Gamma|^N}=1$, or $\det M = |\Gamma|^N$. Note that in this case all matrices from the group act as identity operators (since the whole space consists of invariants), thus they are identity operators, and the group is actually trivial: $|\Gamma|=1$ and $\det M=1$.
- The subspace of invariants is proper. Then, the projection operator has a nontrivial kernel and is not injective, so $\det\frac{M}{|\Gamma|}=0$ and $\det M=0$.
Thus, we have only two options:
- $\Gamma$ is trivial and $\det M=1$
- $\Gamma$ is not trivial and $\det M=0$
Note that in any case $\det M \in \{0,1\}$, not just $\det M \in \mathbb{Z}$.
Hint:
Let $\Gamma=\{A_1,\ldots,A_n\}$.
Note $$M^2 = \sum_{i,j} A_i A_j = n \sum_i A_i = n M.$$ (For fixed $i$, $\Gamma=\{A_iA_1,A_iA_2,\ldots,A_iA_n\}$, so the double sum goes over all elements, each $n$ times.)