Why doesn't this function terminate in Haskell?

By typing the function on two separate REPL lines, you are essentially redefining it the second time around, omitting the base case.

The correct way to enter this function into the REPL is:

nest f 0 = id; nest f n = f . nest f (n - 1)

Alternatively, you can enter multiline mode with the :{ command, and leave it using :}.

When you pasted it into GHCi what you did was define one function of nest f 0 = id. Then you said "ignore that function, I'm replacing it with a new function of the same name where the whole definition is nest f n = f . nest f (n - 1).