A paper review I've undertaken turning out to be harder than expected
Is it common for a reviewer to review only a part of the paper, and leave the rest of the paper to other reviewers?
I have done that on at least two occasions. Sometimes it is the only honest option.
On the other hand: is it my duty, as a reviewer, to complete my review, regardless of how much time it takes?
Not really. It is commonly considered a duty of a career mathematician to contribute to peer review of mathematical works (at least) proportionally to their own publishing. Whether you achieve this by refereeing some really tough papers or a lot of simple ones is up to you.
Consult the editor
Adding to @darijgrinberg's answer: Consult with the journal's editor(s) before making any decisions.
They might:
- Tell you more about what's common in a situation such as the one you're in.
- Possibly decide they want to switch reviewers or divvy up the review work
- Tell you what they expect from you (including telling you or hinting at you what they would regard as unethical).
- Pressure you into continuing (not such a great outcome - but I have to add this in fairness)
Don't feel awkward or inappropriate about taking this up with the editors, it's the responsible thing to do.