Interesting recursive behaviour with \frac error

To understand what is going on, let us consider an example that is a bit simpler. Define \macro to be a macro taking one argument

\newcommand\macro[1]{abc}

The argument is ignored and the expansion only contains abc. Now let's use \macro in \text but omit the argument.

\[ \text{$\macro$} \]

The result is similar to what you observed:

To further understand this let us write out something that is similar to what \text does under the hood (radically simplified of course). The output is the same as with \text.

\[ \mathchoice{\hbox{$\displaystyle \macro$}}%
              {\hbox{$\textstyle \macro$}}%
              {\hbox{$\scriptstyle \macro$}}%
              {\hbox{$\scriptscriptstyle \macro$}} \]

Here we used \mathchoice which is a primitive which takes four arguments, one for every possible math style, indicated by the style macros that I'm using inside the \hbox. Each argument is expanded and typeset in a temporary box but only the one corresponding to the current style is going to be output. So the following snippet would result in

\[ \mathchoice{D}{T}{S}{SS} \] % -> D
$ \mathchoice{D}{T}{S}{SS} $ % -> T
$ _{\mathchoice{D}{T}{S}{SS}} $ % -> S
$ _{_{\mathchoice{D}{T}{S}{SS}}} $ % -> SS

This is the mechanism that makes \text adapt correctly to the surrounding math style. Now what happens in our example of the missing argument. If we expand \macro we get

\[ \mathchoice{\hbox{$\displaystyle abc}}%
              {\hbox{$\textstyle abc}}%
              {\hbox{$\scriptstyle abc}}%
              {\hbox{$\scriptscriptstyle abc}} \]

Oops, \macro ate the closing $ and now there are lots of mismatched $. We can now follow TeX as it is scanning along by reading the error messages. The first error is

! Extra }, or forgotten $.
l.4 \[ \mathchoice{\hbox{$\displaystyle \macro$}
                                               }%

TeX tells us that the error occurred when reading the } after \macro$, indicated by the linebreak. It also tells us that it was expecting a $ but instead encountered a }. Fair enough, since \macro just swallowed the $. The exact same happens in the following lines and TeX is still looking for that closing $ that never comes, so everything is being recorded in the first argument to \mathchoice. This even extends beyond the math environment and only ends once the $ are balanced or the end of the document is reached. Eventually TeX realizes that things have been going wrong and it starts inserting multiple $ and } until it is satisfied. Then it typesets the first argument of \mathchoice which contains everything up until the invalid part. You can produce fun stuff with this

\documentclass{article}
\usepackage{xcolor}
\newcommand\macro[1]{abc}
\begin{document}

$ \mathchoice{unused}%
             {\hbox{\color{red}$\textstyle \macro$}}%
             {scriptstyle}%
             {scriptscriptstyle} $
Wait, what is happing?
Why is everything red and in one line?
What happened to the linebreaks?
When is this ever going to end?

\end{document}

With \frac the situation is a bit more difficult to explain because it has a group inside of it but the rough working principle is the same. The unterminated macro eats the trailing $ and TeX keeps on scanning until the $ are balanced or the end of the document is reached.

The premise is wrong. There is a second argument to \frac in your code, which is precisely $.

The definition of frac is

\DeclareRobustCommand{\frac}[2]{{\begingroup#1\endgroup\@@over#2}}

where \@@over is the same as the primitive \over that amsmath redefines to yield a warning. Since the command wants two arguments, it can scan {123} as the first one and then, since no { follows, it absorbs the next token as the second argument. Thus you get the same as

$\frac{123}{$}

Now this is transformed into

${\begingroup123\endgroup\@@over$}

and this already will produce

! Missing } inserted.
<inserted text>
                }
l.6 $\frac{123}$

?
! Too many }'s.
\frac  #1#2->{\begingroup #1\endgroup \@@over #2}

l.6 $\frac{123}$

?

even if not inside \text. The $ token is seen where it should not be, because of the yet unbalanced {, so TeX adds } for recovering; then it rescans $, which ends math mode and the next } triggers an error because it is no longer balanced by a corresponding {.

Within \text the situation is even worse, because \text will produce four versions of its argument in order to use just one of them according to the math style it ends up in. The two errors above will become eight error messages, but slightly different, because they're produced when TeX is in the middle of working on \text (\text@ is its internal version, just for information):

! Missing } inserted.
<inserted text>
                }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Missing { inserted.
<to be read again>
                   }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Missing } inserted.
<inserted text>
                }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Missing { inserted.
<to be read again>
                   }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Missing } inserted.
<inserted text>
                }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Missing } inserted.
<inserted text>
                }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Extra }, or forgotten $.
\text@ ...style \ssf@size {\firstchoice@false #1}}
                                                  \check@mathfonts }
l.6 \[\text{$\frac{123}$}
                         \]
?
! Extra }, or forgotten $.
\text@ ...firstchoice@false #1}}\check@mathfonts }

l.6 \[\text{$\frac{123}$}
                         \]
?

The three variations of “abc” followed by a denominatorless fraction result just as a byproduct of the error recovery and they shouldn't be taken “seriously”. It's actually quite surprising that hitting return eight times gets TeX in sync.