What mode does the terminal go into when I type a single quote?

The shell is just waiting for the closing quote. When you enter it, it will do exactly what it usually does, and attempt to execute the command entered.

Quotes cause the shell not to interpret special characters, meaning that expansions are not performed. Single quotes suppress all interpretation of special characters completely. Normally a newline separates commands, but here you have included the newlines as part of the command by quoting them.

Since there is no such command as <newline>ls<newline>, it is not found.

Effectively, the shell asks for a complete command/expression, and for that reason is displaying the PS2 prompt string.

From man bash:

PROMPTING

When executing interactively, bash displays the primary prompt PS1 when it is ready to read a command, and the secondary prompt PS2 when it needs more input to complete a command.

And a little before that:

  PS2    The value of this parameter is  expanded
          as  with  PS1  and used as the secondary
          prompt string.  The default is ``> ''.

Thus, as you may guess from reading the documentation, shells have multiple prompts with different purposes. The PS1 prompt is your root@sai:~# string, which shows up normally when you enter commands. > is the PS2 prompt. There's others, too: PS3 for select command block and PS4 for debugging with set -x command. In this case we're more interested in PS2.

There are many ways in which shell may show the PS2 prompt (and where completing a command on a new line might be necessary). The same prompt is used when you perform here-doc redirection (where a command is considered complete when you see the terminating string, in this example, EOF):

$ cat <<EOF
> line one
> line two
> EOF
line one
line two

Very often continuation of a lengthy command can be done by adding \ and immediate(!) newline, which will cause the same prompt to appear:

$ echo Hello\
> World
HelloWorld

$ echo 'Hello\                                                                                                           
> World'
Hello\
World

When pipes, logic operators, or special keywords appear on command-line before newline, the command also is considered incomplete until all final statements are entered:

$ echo Hello World | 
> wc -l
1

$ echo Hello World &&                                                                                                    
> echo "!"
Hello World
!

$ for i in $(seq 1 3); do
> echo "$i"
> done
1
2
3 

$ if [ -f /etc/passwd ]
> then
>     echo "YES"
> fi
YES

In your particular case, a single quote implies literal interpretation of what is between the single quotes. Thus, as Zanna pointed out, you are entering a command that consists of newline+ls+newline. Such an executable filename cannot be found (and typically command filenames should consist of only alphanumeric characters, plus underscores, dashes, and dots). Although it is indeed possible to have filenames that contain special characters in them, it is always avoided.

NOTE: such behavior as shown in your example is specific to Bourne-like shells, including bash, dash (on Ubuntu it is symlinked to /bin/sh), ksh, and mksh. csh and its derivatives do not behave in such way:

$ tcsh                                                    
eagle:~> '
Unmatched '.
eagle:~> csh
% '
Unmatched '.
%  

However, in interactive mode, csh will still raise ? as prompt2 when more input is required:

$ csh
% foreach n ( 1 2 3 )
? echo $n
? end
1
2
3  

See also:

• In which situations are PS2, PS3, PS4 used as the prompt?
• What's the difference between <<, <<< and < < in bash?
• What does > mean in the terminal!
• Unable to enter new commands in terminal — stuck with “>”
• What is the effect of a lone backtick at the end of a command line?