What happens if a math PhD student fails to find a proof that is the main objective of his/her thesis topic?

I wrote a failed dissertation myself. No papers resulted from it. Fortunately, I had a significant side project which was published, and 11 years later, the main thread of my research still comes from this side project. (I suppose I could have written up the side project as my dissertation instead, but I didn't.)

First, no project is ever a complete failure. At the very least, you manage to prove some (possibly almost trivial) special cases, you prove some helpful lemmas, you find that certain lemmas you might have hoped would be helpful have counterexamples, and you find certain methods that you might use don't apply because some hypotheses needed to use those methods aren't satisfied in your case. It's possible to write all of these things up, ending up with a dissertation that's basically about "How not to solve Problem X (except in this very tiny special case)". (This is basically my dissertation.)

Second, it's rare for mathematics graduate students, especially at top programs, to work on only one problem. Your advisor might suggest a main problem for you to work on, but you go to seminars and hear about other problems, talk to other graduate students or postdocs and learn about other problems, and so on, and graduate students are generally encouraged to spend at least a little time thinking about these other problems. If you get stuck on your main problem, you still have other problems to solve, and it's quite common that what you learned to work on your main problem ends up helping you in solving these other problems instead. (This is basically what happened to me, except this other problem didn't end up in my dissertation.)

Third, especially in the early stages of working on a problem, advisors are usually fairly quick at pulling the hook if it looks like no progress is being made. Most advisors know of lots of problems, and they know what complete lack of progress due to a problem being too hard looks like. Even later on, advisors can sometimes suggest simpler problems that can be solved in a shorter time frame (given what a student has already learned). In some cases, after a few rounds of failed problems, the student ends up with a dissertation that's about as weighty as a half-decent undergraduate research project (and results in zero papers or one paper in a "write-only" journal).

It's true that a dissertation written out of a failed or almost trivial project tends not to bode well in applications for jobs where research matters (unless there is a more substantial side project). Sometimes an influential or convincing advisor can make a strong enough case in recommendation letters for the student to get a postdoc, but this is harder now then it was 10 years ago given how much more competitive the job market is.

A good advisor will steer students away from "all or nothing" problems, the sort of problems where complete failure is a realistic possibility. For example, one stereotypical case is an elementary attempt to resolve a famous question in number theory, where the most likely outcome is a concrete understanding of why this specific approach just can't work, and nobody else will be terribly interested since they never thought it had a chance in the first place.

A good thesis problem needs to have several properties:

1. It should be interesting and attention-getting if solved. This is the easy property to achieve.

2. It should be in a rich and diverse enough area that any serious attempt to solve it will uncover something worthwhile in its own right, even if it doesn't lead to a solution of the original problem.

3. It should help the student build knowledge and prepare to branch off in several new directions. (I.e., you shouldn't reach the state of being done with nothing more to do, and continuing work should naturally grow broader rather than narrower.)

In particular, #1 is far from enough by itself. If you have #2 as well, then it doesn't really matter whether the original problem is solved, while #3 is important for setting the student up for success beyond graduate school.

So the optimistic answer to your question is that the advisor should take care to prevent this from becoming a problem, by guiding the student to a problem where failure to solve it is not a disaster. Of course there's also the pessimistic scenario in which the advisor screwed up or the student wouldn't take advice, and the problem really isn't suitable for a thesis. In that case you have to muddle through as best you can, probably by writing a suboptimal thesis and then trying to make up for it by other work afterwards. Fortunately this scenario doesn't seem to occur all that often. (And the worst case of all is when the student just isn't accomplishing anything, but that can happen in any field.)

There are a number of successful outcomes that don't involve finding that desired proof. Here are some examples:

1. Very often, a good mathematical question is good not simply because the answer would be useful, but because the techniques that might lead to an answer are useful. If you develop new mathematical tools, those might form a worthy thesis in themselves even if they don't amount to a proof.
2. You may achieve some intermediate result that opens a new possibility to prove the bigger result. Indeed, many theses only aim for this if the bigger result is something huge like the Riemann hypothesis, the BSD conjecture, etc.
3. In searching for a proof of X, you may discover a related question Y and find a proof that resolves Y. Y might be more interesting than X, or simply more approachable. This is discussed in Uri Alon's well-known short paper on choosing a problem; I recommend that paper to anyone who is asking this question.

A good advisor will help you find a thesis topic that has multiple avenues and multiple stages of potential progress, so that it is not an all-or-nothing venture.