How do I switch between Chrome windows while in full screen on a Mac?

Short answer: not possible.

This is not possible because the way in which Mac Spaces work and the function you're using are incompatible.

Mac Spaces organise content in different Spaces, and when you go fullscreen, that window gets its own Space.

cmd + ` switches between windows of the same application, but this state is not shared between Spaces.

Control+→ and Control+← is a default keybinding to switch between full-screen applications on MacOS.

As far as I know, there is not a way to limit it to only Google Chrome.