Why aren't representations of monoids studied so much?
Whenever you see people using Young tableaux to discuss representations of $GL_n$ they include an apologetic "we'll only consider polynomial representations, i.e. not $det^{-1}$". That is to say, they're really thinking about representations of the monoid $M_n$. But monoid representations (and monoids) aren't as fashionable as group representations, I guess. This case is special too in that $GL_n$ is dense in $M_n$, so the group rep determines the monoid rep.
Why do they want $M_n$ reps rather than $GL_n$? Because they really want reps of the whole category $\bf Vec$, i.e. functors ${\bf Vec} \to {\bf Vec}$, like "alt square". Then one can restrict such a functor to the single object ${\mathbb C}^n$ and its endomorphisms, obtaining a rep of $M_n$, and then further restrict to $GL_n$.
Certainly irreducible representations exist; one can still construct the monoid algebra of a monoid and consider modules over the algebra. But Maschke's theorem is false in general for finite monoids. Indeed, consider the monoid $M = \langle x | x^3 = x^2 \rangle$. Complex (for the sake of argument) representations of $M$ are the same as representations of the monoid algebra $\mathbb{C}[x]/(x^3 - x^2)$ and this algebra is not semisimple, so its finite-dimensional representations are not completely reducible. (Note that the usual proof of Maschke's theorem fails miserably; you can't average inner products over a monoid without inverses, and any unitary representation of a monoid has to factor through the Grothendieck group).
So part of the answer may just be that the representation theory of monoids is inherently more complicated. Although this isn't doesn't seem to be stressed much in textbooks, having inverses is a pretty important structural property of groups; it endows group algebras with an antipode and endows the category of representations of a group with duals.
The representation theory of finite semigroups is an interesting blend of group representation theory and the representation theory of finite dimensional algebras. The subject is both old, going back to A.H. Clifford (from Clifford theory in group representation theory), and at the same time is in its infancy.
The reason why semigroup representation theory is not so well studied, in my opinion, lies in its origins. A description of the simple modules for a finite semigroup was given by work of Clifford, Munn and Ponizovsky in the forties and fifties. It was further clarified by Rhodes and Zalcstein and by Lallement and Petrich in the sixties. Roughly speaking the main theorem states that all irreducible representations of a finite semigroup can be constructed from irreducible representations of associated finite groups in a very explicit way. Sadly, this beautiful work was written up using very heavily the structure theory of finite semigroups, which is not widely known, and so the literature is virtually inaccessible to nonspecialists. The approach used here foreshadows the development of stratified and quasihereditary algebras by Cline, Parshall and Scott. In fact, in 1972, Nico computed a bound on the global dimension of the algebra of a finite von Neumann regula semigroup by finding a sequence of heredity ideals and discovering the bound it gives on global dimension years before the notion of heredity ideal was invented.
The character table of a finite semigroup was investigated in the sixties and seventies and shown to be invertible (although it is not orthogonal like in the group case). A method for writing a class function as a linear combination of irreducible characters was given that amalgamated the group situation with Möbius inversion in posets.
Progress on finite semigroup representation theory then more of less stalled for a number of years. I believe this was for two main reasons.
- There was a lack of ready-made applications.
- Semigroup algebras are almost never semisimple and the modern representation theory of quivers, etc. were only invented on the seventies. By then finite semigroup theorists were interested in other problems and they were mostly unaware of developments in the representation theory of finite dimensional algebras.
In the eighties and early nineties, there was some renewed investigation of the representation theory of finite semigroups, mostly due to Putcha, Okninski, Renner and their collaborators. In particular, connections with quivers and quasihereditary algebras and other aspects of modern representation theory were made.
The past decade has seen a renaissance in the subject of semigroup representation theory, spurred on by probabilists and algebraic combinatorialists. Bidigare, Hanlon, Rockmore, Diaconis and Brown, to name a few, have shown that a number of random walks are much more easily analyzed using semigroup representation than using group representation theory. For instance, it is nearly trivial to compute eigenvalues for the riffle shuffle and the top-to-random shuffle using semigroup theory. It is more difficult to use the representation theory of the symmetric group. Moreover, the diagonalizabilty of these walks is not explained by group theory, but it is explained by semigroup theory.
Also Bidigare's observation that Solomon's descent algebra associated to a finite Coxeter group is a subalgebra of a hyperplane face semigroup algebra has been important to people in algebraic combinatorics. There are also applications of semigroup theory to automata theory in particular in connection to the notorious Cerny conjecture on synchronizing automata.
In the last year, a half-dozen papers on semigroup representation theory have appeared on the ArXiv, many by nonsemigroup theorists. I expect that the trend will continue. We now know how to compute the quiver for a large class of finite semigroups, describe in semigroup theoretic terms projective indecomposable modules and for some classes we have techniques for computing global dimension. Semigroups with basic algebras over a given field have been described.
What is needed is a book covering all this for the general public!
Edit. Our new paper gives a close connection between monoid representation theory, poset topology and Leray numbers of simplicial complexes with classifying spaces of small categories thrown in. If browsing this paper doesn't convince you that monoid representation theory has something to it, then I don't know what will.
Edit. (2/18/14) Since this question just got bumped, let me add the new paper http://arxiv.org/abs/1401.4250 which gives a general introduction to Markov chains and semigroup representation theory and new examples.
Edit(4/1/15). Since this question just got bumped again, let me add that I am in the process of writing a book on the representation theory of monoids. In a sense I started writing this book because of this question (which was the first MO question I ever answered). Hopefully the book will be an answer to this question. I will make a link available shortly from my blog page.
Edit. (Question was bumped again) The book is published, and called "Representation theory of finite monoids." Link to the book at publisher's page