Plagiarism checker for mathematics paper

As in jakebeal's comment, if there is actual plagiarism, it will be picked up in the text, so you could use a generic plagiarism checker. Note the arXiv checks for text overlap and picks up quite a few things (the stated purpose is not to catch plagiarism, but it still does, though mostly "self-plagiarism" as far as I am aware).

That said, having done a fair amount of reviewing in math, this is not something that's typically done. Ideally, as a reviewer, you should be fairly aware of what's already been done in the area, and at the least, literate enough in the area to search for what has been done. (One tip: look at the crucial references, and check for papers citing them.)

What is much more likely than outright plagiarism is that some or all of the main results are already published, but unknown to the author. This may be because the results are not stated in the exact form the author is proving, or just the author wasn't aware of the appropriate papers. (I once pointed out that one of the papers the author referenced contained stronger results than the author proved--the author just didn't realize it!)

In any case, a proper literature search in the process of reviewing will turn up plagiarism almost all the time if it happens, and moreover is important for a good review. So my advice is don't worry about using a plagiarism checker, just do your homework.

In the journals I help edit, admittedly not in the mathematics field, a plagiarism check is one of the first internal checks undertaken on a submission. I would be surprised if there isn't one in place in most journals. The results of the check are forwarded to the deputy editor for consideration. It is not released to the reviewers. Perhaps your journal has access to math-specific checkers.

If you're worried about a particular manuscript, I suggest requesting a plagiarism check through the deputy editor who's handling the manuscript. You don't need to wait for the results before proceeding with review. I suggest incorporating your request within your review itself. For example:

Review of Manuscript 16.10634.1

Numerous typographical errors in the abstract.

I recall seeing part of the proof somewhere. Please perform a plagiarism check.

Etc.