How do I get out of a >> prompt in powershell when I made a mistake?

Press ctrl+C.

@jon Z 's answer tells you how to abort it, but what you want is to hit enter at the empty >> prompt and you will be out:

PS > git commit -m 'test
>> '
>>
[temp 7b96875] test
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 c

Notice in the last >> prompt I just hit enter and it completes the command. This is done when you want the command multiline ( in this case commit message is multiline) and to signify the end, you just hit return on an empty line to get out.