After successfully publishing papers during my Post-Doc, why am I now having trouble publishing as a tenure track academic?

I feel for you. I have been in the (pure math) game for a lot longer than you, and sometimes my work is still treated shabbily by editors and referees in a way that is hard to figure out. I feel that the publication process has gotten more opaque in the dozen or so years I've been involved with it: at the beginning of my career, whenever a paper got rejected, I would come away with a clear reason for why. I have had some truly head-scratching rejections in recent years...along with acceptances in prestigious journals, which is good for me but even more confusing in this regard.

I think that the best advice I can give to you is to find a trusted senior mentor -- someone who understands the content of your work and the sociology of your field -- and really go over your situation with them in minute detail. I would certainly go over all of the referee reports in detail. If any one rejection feels especially unexplained to you, please realize that you certainly can ask the editor for more information, if you do so extremely politely and while making clear in advance that you are not arguing but rather asking for more information. (I have experienced the wildest of extremes in response to such inquiries: I have gotten no response at all, and I have also gotten "I'm so sorry, your paper was rejected by mistake! Would you like to resubmit it?" Seriously.)

Without this specific information, we can just guess on top of your guesses. But that's interesting to me, so I'll play along. First, here is one piece of information that probably should have come before the end of your question:

My goal is to just get these papers published; at my current institution, the publication of these two papers alone would be enough to satisfy all of the research requirement for tenure.

This is pretty key. Before I read it, I was going to respond to your worries that you are publishing work in a very narrow field that is (perhaps!) being viewed by the same small set of people as unpublishable by suggesting that you make an effort to branch out a bit in your work. This is what everyone eventually does anyway. But if your goal really is to only publish the work you've already written up, that's quite different. In particular you talk about how you have a very high teaching load (33 credits per year = more than five courses per semester?!? I hope I got that calculation wrong) and thus don't have time to do research. Well, you may have zero time for research during the academic year, but if you need two publications for tenure you can spend entire summers working on publishing these papers...and you probably should, if you think your tenure depends on it. I note though that you said you needed only these two publications; you didn't say that you wanted to stop writing and publishing papers after that.

Okay, now to examine your reasons:

They are just not up to shape, because I don't have the time to focus on polishing them.

As I said, I think you should make the time, but that doesn't invalidate the suggestion that a lack of polish is why the papers haven't been accepted as yet. This is where you should get help from mentors and colleagues and look at the referee reports. Based on the information given, this explanation sounds a bit unlikely to me: you've already published five papers in a very similar field, so you know what you're doing here. Math research papers really should not be rejected for lack of polish; they should be rejected when the writing is so bad that the referees can't make sense of them, or when the writing is bad and the results are also not up to snuff.

The referees hate me. This may seem silly, but...

Actually you sold that explanation pretty well to me: if you had a terrible falling out with a key person in a tiny field and one other known past referee has told you that you're going in a wrong direction (to weigh in on comments about that: yeah, I think that's a nasty thing to say to someone), then it may well be that your work is getting evaluated by the same small number of people again and again and they just don't like it. If that is the case, I would again recommend branching out a bit...perhaps even without starting a new project. Most mathematical work can be cast in various ways and in order to appeal to various people. I find it hard to believe that there are really on the order of 4 people in the world who are interested, but even if that's true your job is then to interest other people. You could try for instance rewriting one of your papers to be a lot more explanatory and appeal to a much more general mathematical audience, and then send it to a not-too-exalted generalist journal. If you do this, there's a very good chance that an editor who is "out of the loop" is not going to send the paper to the same small, sour group of people.

Chance. The reviewers of the first papers happened to be interested in those topics, and the reviewers of the current papers have not been interested.

Yes, it certainly could just be chance, and in my experience chance plays a distressingly large role in contemporary mathematical publishing. However, even if it is just chance, you can still probably improve the situation by changing it in some way.

Good luck.

Do your tenure requirements list specific journals you have to publish in? It would seem reasonable to submit to the "least prestigeous" journals that still satisfy the requirements.

Also, some journals allow you to exclude potential referees. Maybe it helps to take the two suspects out of the picture?