Level of difficulty of exams in math PhD and anxiety over being able to do well

This depends very much on your university and your professor. There is a whole range of opportunities spanning a wide range of difficulties. This issue alone shouldn't be the reason to drop the idea of the PhD degree itself.

If you are really interested in the field and are willing to work for it, then by all means, continue; don't let a little test put you off. PhD is not about exam scores, it is about research quality and how much you could contribute to your field. If you are able to hold your ground on that, stay strong and move on.

My best wishes!

On our midterm the highest grade was a 35 percent. I was told this is how nearly all of the exams this professor gives turn out. Is this a common thing to happen? I suddenly feel inadequate to be a phd student in math.

Why?

As others have pointed out, most PhD students spend some of their time while doing a PhD worrying about whether they belong. (And this is not a feeling that vanishes upon getting a PhD.) There's nothing wrong or worrying about feeling that way.

But, to the extent that feelings are amenable to rational inquiry, this doesn't seem like a good reason to feel that way. Raw percentages on exams are fundamentally arbitrary: they can be arbitrarily altered, independently of how much the students know, simply be asking easier or harder questions, or more or fewer questions. The raw information "I got X% on the exam" tells you exactly nothing.

Undergrad classes tend to follow some conventions, in part because students react negatively to low percentages anyway. This involves writing exams with some fluff that almost everyone gets, which brings the average up. Grad classes do less of this, in part because there's a higher expectation that the students will be able to interpret the score with more maturity (though it sounds like your professor has taken this to an unusual extreme).

So, if you're worried about what the score means, by all means investigate what it means. But the question you should be asking (possibly of your professor) is "what scores are promising/acceptable/worrying?" And, even then, don't overreact to one score in one class, especially when you have other scores from the same class.

Since you get high homework grades, I guess that there is little reason to believe that you are entirely unfit for a PhD in math. To the contrary, doing a PhD is about learning to do research, and research is a process that takes place on time-scales of the order of weeks to years. So, getting low grades in an exam situation, where you are pressured into producing certain answers in a very short time span and unnatural setting, is hardly a tool that reliably assesses your ability to do research. Performance on homework, where you have more time, acccess to resources, and freedom to let your mind do its thing, is a much better metric.

The above is, I would claim, in almost all cases true. Given that the best student (and I assume there is more than just a couple) has achieved only 35 percent on the exam, I would worry even less about your performance on the exam. And given that it is said that this is a common occurrence with this professor, there is even less reason for you to worry. Some professors just give bad exams, i.e. ones that are not adapted to the course, reasonable expectations of ability of students, and the circumstances the exam is given in, time being an important example.

By the looks of it, this might well be one those professors. And since, I assume, he doesn't fail almost entire classes year after year, I would not worry that you would fail if you keep up your good work, and not worry about you being unable to pursue your PhD.