How do you check that your mathematical research topic is original?
(1) It depends a lot on the field. In fields that rely on specialized techniques discovered relatively recently or known only to a few, or fields where the questions involve recently-introduced objects, it's much easier to keep abreast of current research.
On the other hand, in fields with elementary questions that could have been studied a hundred years ago, sometimes even senior mathematicians discover that their work was studied a hundred years ago.
Of course, working in a trendy field carries its own risk, that someone else could be working on the same thing at the same time, but not much can be done about that.
(2) If you're working in a specialized field, as other have said, the best thing is to ask your advisor. If you have an advisor in a specialized field and have ideas in a different field, the best thing would be to ask someone in that field. As a grad student you probably want to start with fellow grad students, but a senior mathematician would probably asks someone on their own level.
If you have an idea that is more elementary, you should still ask your advisor, but there are certain mathematicians who know a lot of elementary and classical mathematics you could potentially ask.
(3) With regards to literature review, one trick that helps a bit when keyword searches fail is to use citations. If your idea generalizes work of Paper X, or answers a question from Paper X, or uses in a fundamental way the results of Paper X, anyone else who had the same idea would likely cite Paper X. You can produce a list of papers citing Paper X on both Google and MathSciNet.
(4) As a starting graduate student, even if your idea is completely new and original, it is likely that the greatest value it provides to you will be as practice for your future work. (I mean if you're good enough to do groundbreaking work right off the bat, you will probably do even more groundbreaking work once you get some experience under your belt.)
So don't feel bad at all if you find out something was already well-known - the experience of formulating and solving your own problem makes you well-placed to do original research once you learn a bit more, as compared to someone who knows a lot but hasn't done this.
As was echoed in the comments by YCor and Mikhail Borovoi, the question of originality (especially for results that could have been stated a long time ago) is one that is relevant to all mathematicians. An interesting recent example that comes to mind is this story on Terence Tao's blog.
So how can you guard yourself from this? I think that there are two tools you can use:
Experience. When you're around in an area for a few years, you should develop a pretty good sense of the literature. This is where an advisor can be really helpful when you're young, because by definition you don't have much experience at that point.
Experience also means knowing the correct search tools, like looking in papers that cite a given one. But sometimes you just don't know the correct words to search for, so then there is:
- Asking other people. This is related to 1: one of the most useful skills is knowing whom to ask, and this again comes from experience and connections developed over a long period of time.
In both of these your advisor has an advantage because they've been around longer. But as you go on you will find that this becomes easier to do yourself.
Of course research output is not the only type of mathematical productivity, and you can still learn a lot by rediscovering an existing result. But it's incredibly frustrating to find out something you proved is already known (especially when you don't have so many papers yet), because it changes the meaning of the time investment you made.
If you just started your graduate study, you probably have an adviser. If you don't have one yet, try to find one as soon as possible. Anyway, in most universities that I now an adviser is required to defend a PhD.
Then show your result to your adviser. Even if it is from a different area from his/her scientific interests. Adviser will probably make a judgement, or if necessary will know whom to ask and where to look.