Are mathematics REUs absolutely limited to undergraduates?

I quote from the NSF REU guideline (emphasis mine):

Eligible Student Participants: Undergraduate student participants supported with NSF funds in either REU Supplements or REU Sites must be U.S. citizens, U.S. nationals, or permanent residents of the United States. An undergraduate student is a student who is enrolled in a degree program (part-time or full-time) leading to a baccalaureate or associate degree. Students who are transferring from one college or university to another and are enrolled at neither institution during the intervening summer may participate. High school graduates who have been accepted at an undergraduate institution but who have not yet started their undergraduate study are also eligible to participate. Students who have received their bachelor's degrees and are no longer enrolled as undergraduates are generally not eligible to participate. For REU Sites, a significant fraction of the student participants should come from outside the host institution or organization. Some NSF directorates encourage inclusion in the REU program of K-12 teachers of science, technology, engineering, and mathematics. Please contact the appropriate disciplinary program officer for guidance. Within the framework of the basic eligibility guidelines outlined here, most REU Sites and Supplements further define recruitment and selection criteria, based on the nature of the particular research and other factors.

So this means that (a) your host will not be able to receive funding for you from the NSF if s/he accepts you into the program (b) on the other hand if you and/or the PI of the grant is able to secure funding otherwise, there's generally no rule saying that a student in your position cannot be involved in research in some way.

You will need to individually contact the REUs you are interested to find out whether they'd be willing to grant you the leeway.

There do exist programs for this period of time (between undergraduate and graduate programs). The ones that comes to mind are EDGE (which you can see on their webpage is limited to women) and the PCMI summer school. I believe there are others, as Henry points out one should google 'summer school' instead of 'REU' to get at them.

REUs are almost all a specific NSF program which, as Willie points out, generally doesn't allow students in your situation.

However the programs called "summer schools" (not to be confused with credit bearing courses offered by universities in the summer) often do allow graduate students (and some have graduate students as their main target).