What happens when you exceed MSMQ journal limit?

Yes, journaling will cease to work for new journal messages. Journal messages should be processed just like any other message so they shouldn't be allowed to accumulate.

I found reports that full journal can even block receiving messages, not just moving them to journal:

https://social.msdn.microsoft.com/Forums/ie/en-US/58e12d5a-acf3-4c67-a02a-9fa7528a5912/msmq-journal-full

However that might happened because MSMQ reached some other quota at that point, perhaps its global storage limit for all queues, and not just because of journal. I couldn't reproduce it just by limiting journal only. But it's anyway good idea not to let journal grow too much.

Btw. we made commercial tool which can periodically backup and empty journal, among other things - QueueMonitor