Would it be weird to give a github version of a paper to a reviewer?

Weird? I don't know. Unusual? Certainly. I have never heard of this, and I would be rather unsettled if it happened to me as a reviewer.

Useful? There lies the rub.

• First of all, this is only useful if the reviewer understands how revision control systems work. You can only be sure of that in very computer-centered fields, and even then, there's no absolute guarantee. Since you said "a mathematical field", I can assure you, based on my experience most people don't know or even care about RCSs. So you need to write an accompanying letter outlining the changes which can stand on its own. And if you do, there's little point in including a link to the github repo.
• Second of all, I really doubt that seeing a diff of a source file written by someone else is really useful. Especially if it involves crazy diagrams or things like this. Reading a long source tex file from start to finish is not comfortable (ask yourself: when you reread your own papers, do you read the source or the PDF?). This introduces a high cognitive load of mentally parsing all the macros, either standard or your own, and turning it into math. It's definitely not something that you want to bother to do when you're doing volunteer work.
• The diffs will also be polluted by fixing the inevitable typos, whitespace changes, and so on. If you do a big reorganization of the paper such as moving paragraphs, the diffs will also not really be useful.

Consider using other tools, such as latexdiff. And before you complain that the output of latexdiff is not good enough in some cases: in those cases, the git diff would not be good enough either.