I am struggling due to insufficient background in a graduate course and feel like a moron. What do I do?

Sounds like my Physics II course decades back. The effort on that course was greater than the effort on all other courses combined. Little did I know that I took it from Paul Chu, who at the time was a Physics Superstar, and he wasn't going to "just pass anyone".

Somehow I managed to get a B. To me that B was worth more than all the A's I obtained combined.

Hard material is not a sign of you being a moron. Perhaps you're not ready for the course; but, the entire idea of taking a course is to challenge you with new material. The harder the challenge, the more you will learn.

Take it easy on yourself. Try to do as much as possible without over involving your professor, but do use an hour of their "office hours" to go over problems you have difficulty with. Be very well prepared for this hour, showing them what approaches you've tried and failed at, and can they teach you "how to approach the problem" instead of "how to do the work".

For support, find others in your class that seem to be bright enough to warrant working with, and form a study group. If the group starts to look like an unbalanced "share the answers" group, get out of it, as it will do nothing to prepare you for the exams.

Rearrange your schedule for the reality that you'll be spending six to eight hours a day doing this class's work, until you can gain enough mastery to cut it down to a mere four hours a day. Remember, it won't last forever, you only need to keep up the crazy schedule till you get out of the class; and, should you have to repeat the class (which is common for such classes), you'll not be under as much of a burden due to your gained familiarity with the material.

In short, hang in there; but, hang in there smart.

@Edwin suggested you "hang in there". And - maybe you can. But if you're still early in the semester, then maybe that's not the right thing to do. You wrote:

It's clear to me that I didn't have the necessary mathematical physics background going in

I'll assume that's true. That means that either the course doesn't list its prerequisites properly, or doesn't enforce them. This can be a disservice to people who actually need to study the prerequisite material (= most people)...

I'm starting to doubt my ability to do this,

Let's assume that you can't do this right now. Maybe you can, but it's not clear that such a gamble helps you; it certainly doesn't help your emotional well-being, which is important to take care of during your Ph.D., much more so than you might think.

In other words, assume you do need to take a course in Mathematical Physics before taking Galactic Dynamics, or spend time self-studying Mathematical Physics.

but I can't quit.

Why? That is, why can you not un-register for this course, if you realize early enough that you're missing pre-requisite knowledge? It sounds perfectly reasonable. As a teacher I would tell you that it does not reflect poorly on a student when they say "I've noticed I'm missing some prerequisite knowledge/skill"; on the contrary, it is a sign of a more mature self-awareness.

What do I do?

(again, this is advice only for early in the semester.)

• Talk to the course teacher.
• Tell them that you believe you lack "the necessary mathematical physics background"
• Ask them to transfer the registration to the next semester.

Note this approach is more problematic if the course is only given once a year; but even in that case, ask the teacher about another potential arrangement.