Apple - How to exclude Mail messages from Finder search?

It is inconvenient, but if you add -kind:mail after your search term in the Finder, it will exclude Mail messages too.

First of all, I should point out that — despite what it looks like — search in Finder is indeed Spotlight but with more options shown. The following information is based on research I've just done on my Mac running Mavaricks (10.9.2):

The reason why it doesn't work to just uncheck Mail is because (confusingly) the SysPrefs > Spotlight > Search Results tab only applies to the top-right corner Spotlight search and not the one in Finder.

However, there is a work-around that answers 2 of your question and proofs that Spotlight is the engine for Finder searches (and ditto for other apps, including Mail and Outlook):

You can exclude a folder from Spotlight-powered searches by dragging said folder to the list in the SysPrefs > Spotlight > Privacy tab.

Caveat/but/proof:

If you were to do this with the data folder of Mail.app (~/Library/Mail), you'll get this warning:

...Telling you that "the search feature won't work in some applications" If you click OK, Mail messages won't show up in Finder (or Spotlight) anymore but the same will be the case for the internal search feature in Mail.app itself, since it's Spotlight-powered...

(@Aaron Bertrand: Same for Outlook, since it's also Spotlight-powered. See first link above)

With regular folders — i.e. not in your hidden-by-default Library folder — you won't get a warning and hiding'll work with no caveats.

There actually is an option to tell Finder to not seach for a given type by writing "NOT eml" for instance, in the search field, which would exclude e-mail files. If you really wanted to, you could create an Automator-service and bind ⌘+F to it via SysPrefs > Keyboard > Shortcuts. The Automator AppleScript would be akin to:

activate application "Finder"
tell application "System Events"
    keystroke "f" using command down
    keystroke " NOT eml"
    key code 123 using command down #123 = Left Arrow, so we're moving the cursor to the beginning of the field
end tell

(I suck at AppleScript so there might be a cleverer way to go about it)