How do you decide whether a question in abstract algebra is worth studying?

I'm going to interpret your question in the language of Gowers's "two cultures" essay as follows:

How does one get good at theory-building?

The process of developing a good theory can seem deceptively simple. One takes some definitions, perhaps by generalizing some known definitions, and deduces simple consequences of them. In comparison with the work required to solve a hard problem, this seems easy---perhaps too easy. The catch, of course, is the one you raised: there is a significant risk of spending a lot of time studying something that ultimately has very little mathematical value. Of course there is also the risk of wasted effort when trying to solve a specific problem, but in that case, it's at least clear what you were trying to accomplish. In the case of theory-building, the signposts are less clear; maybe you succeeded in proving some things, so your efforts weren't entirely fruitless, but at the same time, how do you know that you actually got somewhere when there was no clear endpoint?

The number one principle that I keep in mind when trying to build a theory is this:

Relentlessly pursue the goal of understanding what's really going on.

I'm reminded of a wonderful sentence that Loring Tu wrote in his May 2006 Notices article on "The Life and Works of Raoul Bott." Tu wrote, "I. M. Singer remarked that in their younger days, whenever they had a mathematical discussion, the most common phrase Bott uttered was “I don't understand,” and that a few months later Bott would emerge with a beautiful paper on precisely the subject he had repeatedly not understood." Von Neumann reportedly said that in mathematics, you don't understand things; you just get used to them. This can be valuable advice to a young mathematician who hasn't yet grasped that the reason we're doing research is precisely that we don't really understand what we're doing. However, the key to theory-building is to insist on thorough understanding, especially of things that are widely regarded as being already understood. Often, such subjects are not really as well understood as others would have you believe. If you start asking probing questions---why are things defined this way and not that way? why doesn't this argument actually prove something more (or maybe it does?)?---you will find surprisingly often that what seems like a very basic question has not really been addressed before.

You asked:

How do you decide whether a generalisation (that you find natural) of an established algebraic concept is worth studying? How convincing does the heuristic "well, X naturally generalises Y and we all know how useful Y is" sound to you?

My reply is that the generalization is worth studying if it helps you understand the original concept better. Perhaps the generalization was obtained by weakening an axiom, and you can now see more clearly that certain theorems hold more generally while others don't, so you get some insight into which specific hypotheses of your original object are needed for which conclusions. The heuristic as you've stated it, on the other hand, doesn't sound too convincing to me. I see too much risk of wandering off into a fruitless direction if you're not firmly grounded in trying to understand your original object better.

Keeping firmly in mind that your goal is a thorough understanding of some particular subject is also important because your efforts will, at least initially, not be greeted with enthusiasm by others. You will appear to be a complete idiot who doesn't understand even very basic things that other people think are obvious. Even when you start getting some fresh insights, they will seem trivial to others, who will claim that they "already knew that" (which they probably did, implicitly if not explicitly). Constantly adjusting definitions also appears to others to be an unproductive use of time. Even if you get to the point where your approach leads to a new and wonderfully clear presentation of the subject, and raises important new questions that nobody thought to ask before, you may not get credit for original thinking. Thus it is important that your internal compass is pointed firmly in the right direction. To repeat: ask yourself, am I driving towards an understanding of what's really going on in this important piece of mathematics? If so, keep at it. If not, then you've lost the thread somewhere along the way.

"How much would you subscribe to the statement that studying questions one finds interesting is something established mathematicians do, while younger ones are better off studying questions that the rest of the community finds interesting?"

Not at all. I don't think anyone, young or old, will find success by working on questions other than those they find interesting. Mathematics is just too difficult for that.

Ideally, everyone should work on problems that are interesting to both themselves and the community. Senior mathematicians have the luxury of working on problems whose interest to the commmunity has not been established.

Dear Alex,

It seems to me that the general question in the background of your query on algebra really is the better one to focus on, in that we can forget about irrelevant details. That is, as you've mentioned, one could be asking the question about motivation and decision in any kind of mathematics, or maybe even life in general. In that form, I can't see much useful to write other than the usual cliches: there are safer investments and riskier ones; most people stick to the former generically with occasional dabbling in the latter, and so on. This, I think, is true regardless of your status. Of course, going back to the corny financial analogy that Peter has kindly referred to, just how risky an investment is depends on how much money you have in the bank. We each just make decisions in as informed a manner as we can.

Having said this, I rather like the following example: Kac-Moody algebras could be considered 'idle' generalizations of finite-dimensional simple Lie algebras. One considers the construction of simple Lie algebras by generators and relations starting from a Cartan matrix. When a positive definiteness condition is dropped from the matrix, one arrives at general Kac-Moody algebras. I'm far from knowledgeable on these things, but I have the impression that the initial definition by Kac and Moody in 1968 really was somewhat just for the sake of it. Perhaps indeed, the main (implicit) justification was that the usual Lie algebras were such successful creatures. Other contributors here can describe with far more fluency than I just how dramatically the situation changed afterwards, accelerating especially in the 80's, as a consequence of the interaction with conformal field theory and string theory. But many of the real experts here seem to be rather young and perhaps regard vertex operator algebras and the like as being just so much bread and butter. However, when I started graduate school in the 1980's, this story of Kac-Moody algebras was still something of a marvel. There must be at least a few other cases involving a rise of comparable magnitude.

Meanwhile, I do hope some expert will comment on this. I fear somewhat that my knowledge of this story is a bit of the fairy-tale version.

Added: In case someone knowledgeable reads this, it would also be nice to get a comment about further generalizations of Kac-Moody algebras. My vague memory is that some naive generalizations have not done so well so far, although I'm not sure what they are. Even if one believes it to be the purview of masters, it's still interesting to ask if there is a pattern to the kind of generalization that ends up being fruitful. Interesting, but probably hopeless.

Maybe I will add one more personal comment, in case it sheds some darkness on the question. I switched between several supervisors while working towards my Ph.D. The longest I stayed was with Igor Frenkel, a well-known expert on many structures of the Kac-Moody type. I received several personal tutorials on vertex operator algebras, where Frenkel expressed his strong belief that these were really fundamental structures, 'certainly more so than, say, Jordan algebras.' I stubbornly refused to share his faith, foolishly, as it turns out (so far).

Added again:

In view of Andrew L.'s question I thought I'd add a few more clarifying remarks.

I explained in the comment below what I meant with the story about vertex operator algebras. Meanwhile, I can't genuinely regret the decision not to work on them because I quite like the mathematics I do now, at least in my own small way. So I think what I had in mind was just the platitude that most decisions in mathematics, like those of life in general, are mixed: you might gain some things and lose others.

To return briefly to the original question, maybe I do have some practical remarks to add. It's obvious stuff, but no one seems to have written it so far on this page. Of course, I'm not in a position to give anyone advice, and your question didn't really ask for it, so you should read this with the usual reservations. (I feel, however, that what I write is an answer to the original question, in some way.)

If you have a strong feeling about a structure or an idea, of course keep thinking about it. But it may take a long time for your ideas to mature, so keep other things going as well, enough to build up a decent publication list. The part of work that belongs to quotidian maintenance is part of the trade, and probably a helpful routine for most people. If you go about it sensibly, it's really not that hard either. As for the truly original idea, I suspect it will be of interest to many people at some point, if you keep at it long enough. Maybe the real difference between starting mathematicians and established ones is the length of time they can afford to invest in a strange idea before feeling like they're running out of money. But by keeping a suitably interesting business going on the side, even a young person can afford to dream. Again, I suppose all this is obvious to you and many other people. But it still is easy to forget in the helter-skelter of life.

By the way, I object a bit to how several people have described this question of community interest as a two-state affair. Obviously, there are many different degrees of interest, even in the work of very famous people.