How to cancel recording a macro in Vim?

As noted by @Trendfischer in his comment, the current behavior of vim is not (anymore?) to empty the register on the first press of q.

Now, if you picked the wrong register, here is what you can do:

• do not quit the recording mode (for now),
• create an empty line,
• paste the register you are overwriting (if you're recording in the q register, it's "qp),
• now quit the recording mode (q)
• select the line you pasted above (without end of line mark: 0v$h; don't use V)
• yank it in the register you have overwritten ("qy if you're using the q register; again: don't use the line wise copy Y).
• now you have your old macro back in the register and you can restart the recording of your new macro in the correct register.

Of course, with a bit of effort, you can rescue both the old and the new macro (but you have to start the process before pressing the second time the q): just paste what you have recorded and yank in the "new"/"right" register.

When you record a macro, the register is emptied first and filled as you go so it's very unlikely that there exists a built-in way to save your previous macro while you are halfway through another recording. A way that doesn't involve writing a wrapper around q, that is.

I and a lot of people use a temporary register (@q, for example) for all our one-off recordings. Maybe you should, too.