what is the difference between a GTK theme, Metacity theme, and Emerald theme?

Metacity is the Gnome 2 window manager - the thing at the top of your windows that has the minimize, maximize, and close buttons. On Ubuntu 11.04 and earlier, you can customize your themes to change just that part of the the theme.

The GTK theme is the overall theme that handles stuff like the panel color, the backgrounds for windows and tabs, how an application will look when it is active vs. inactive, buttons, check-boxes, etc. Most theme packages also include a 'Metacity theme' so that everything looks integrated.

An Emerald theme is like a metacity theme, but it is used when you are using Compiz to handle your window management. Compiz can do more visually interesting things than regular theming.

Note that Gnome 3 uses "mutter" as its window manager which is Metacity implemented using the Clutter toolkit. Metacity + clutter = Mutter. :)

What you are referring to as a theme is probably a different kind of theme (in light of your comment on Uri's answer).

Those themes control GTK, Metacity (the Gnome 2 windows manager), Icons, cursors, and sometimes wallpapers all in one. However, they do not control your emerald theme. What you are seeing referred to as a GTK, Metacity, Cursor, or Icon theme is what you get when you click customize.
Each of these controls a different aspect of your interface, and of course, they are all themes in their own right.

In simple terms, Metacity is the GNOME 2 Window Manager, which is sometimes replaced with the Emerald Window Manager via Compiz.

Emerald works on GNOME, KDE, XFCE and more.

A GTK theme, changes the window appearance (colors), not the window borders.