How negative to go in a statement of purpose?

In my opinion you should not "go negative" at all in a statement of purpose. A vivid description of what turned you off to a previous program seems only to give those considering your admissions case something concrete to worry about, whereas just saying that you lost interest and have since regained it (and then talking extensively about the second part!) is not such a hard sell: by the way, most people lack the perspective to find graduate level mathematics (or whatever) interesting. But a certain percentage of students who do an undergraduate degree in mathematics (or whatever) slowly but surely find their interest and passion in the subject increasing until the point where they are motivated to go back to school for further study. As long as you sound interested and motivated now, how you felt in the past is not so relevant.

It may also be helpful to know that your specific story did have a mildly negative effect on me.

The problem was the people in the program didn't like my concentration field (lie groups).

Having your heart set on one mathematical field while starting a master's degree is probably a mistake. At best you know something about your initial interest; you can't possibly know whether you are interested in everything else (and even what other things relate to your interest).

All the profs thought their own fields were more interesting, wanted me to focus on those instead (like discrete geometry).

You were surprised by the fact that faculty are interested in their...interests?!? That surprises me. If you have your heart set on Lie theory, why didn't you go somewhere in which Lie theory was a major research focus?

No other students were really interested in my concentration so important classes got canceled, "low enrollment" they said.

Lack of sufficient interest is exactly what causes courses to be cancelled. You didn't expect them to run classes for you and you alone which lie outside the research interests of the faculty, did you?

By the way, when I went to grad school I had my heart set on algebraic number theory...and ended up doing that (or one of many things that comes under that broad banner). In order to do algebraic number theory I had to learn some Lie theory and some discrete geometry, and these remain of interest to me to this day. Though I know relatively little in each of these areas, I know more than enough to know that they are fascinating fields, and that if I had studied them as a graduate student rather than algebraic number theory per se...well, that would have been fine. And of course my point is not that these two subfields (which I gather are not the "real ones" occurring in your narrative) are so specifically great: rather, believe that if you have the temperament for graduate study in mathematics, then you may just as well study any subfield -- or any subfield of interest to the faculty and your peers in a strong graduate program. So to me, your claim that you would have done well the first time if only you had been learning "the right mathematics" is a bit suspect. It might be true, but it might equally well be true that you just weren't into your studies as much as you needed to be to continue...and now you are. Again, I would go with the latter narrative.

Don't go "negative" in a statement of purpose, or any other statement for that matter. Instead, turn your negatives into positives.

For instance, don't write, "I've had bad experiences with programs that don't offer courses in Lie groups." Just say, "I am very interested in programs like yours that are strong in Lie groups." People know that there are good and bad programs around, but they appreciate your not going negative on others, which saves them the worry of, "when will this guy go negative on us?"

On the other hand, if you cast your aspirations in a positive light, the feeling is more likely to be, "our program is a lot better, or at least more suitable for this candidate than those others that don't emphasize Lie groups." They'd worry much less about what you want that they don't have, because you haven't mentioned those things.