What does yes $(yes yes) do?

It should already be enough to run

 echo $(yes yes)

The $(...) runs the inner command until it is finished and captures all its output. - Now as yes runs a long time and generates a lot of output, bash will eventually run out of memory and crash.

To see the message for the error which closes your shell, you need to run a sub-shell.

$ sh
$ kill -9 $$
Killed
$

$ sh
$ yes $(yes yes)
sh: xrealloc: cannot allocate 18446744071562067968 bytes
$