What is a "modeless" vs a "modal" editor?

A normal, "modeless" editor is like Notepad on Windows: there is only one mode, where you input text.

Vi, and it's successor Vim, are modal: there are two primary modes¹, insert mode where you type text into the editor and it is committed to the document, and normal mode where you enter arguments via the keyboard that perform a variety of functions, including: moving the cursor around the document, searching, and manipulating the text in the document (for example, cutting and pasting).

The Wikipedia article on Vi has a good entry on the modal interface.

The primary appeal, originally a necessity in the early days of Unix computing prior to the widespread adoption of the mouse, is completely keyboard driven editing. This approach has now been more widely adopted in Unix-land, being used for example by a variety of web browsers.

This awesome project, Vim Clutch, provides a clear visualization of the concept of switching between modes.

^{1. There are also two other modes, command mode for entering commands as you would in a shell, and visual mode when selecting text to operate on.}

In user interface design, a mode is a distinct setting within a computer program or any physical machine interface, in which the same user input will produce perceived different results than it would in other settings. The best-known modal interface components are probably the Caps lock and Insert keys on the standard computer keyboard, both of which put the user's typing into a different mode after being pressed, then return it to the regular mode after being re-pressed.

An interface that uses no modes is known as a modeless interface.[1] Modeless interfaces intend to avoid mode errors[2] by making it impossible for the user to commit them.