\varepsilon vs. \epsilon

Historically there has been a lot of confusion over the two forms, (the situation with \phi and \varphi is similar but even more confused as at one point Unicode swapped the reference glyphs). I added a special section about epsilon to the XML/HTML entities spec

http://www.w3.org/2003/entities/2007doc/#epsilon

The situation in TeX is no different really, different communities used different forms of epsilon and it is rather arbitrary which one gets which name. Unicode (now) calls the curly epsilon "GREEK SMALL LETTER EPSILON" (ε) (this is a textual Greek letter rather than a math alphabetic symbol) and the symbol that TeX traditionally assigns to \epsilon is called GREEK LUNATE EPSILON SYMBOL (ϵ) the "symbol" being a hint that this is intended as a mathematical character rather than a textual Greek letter.

---

From Wikipedia:

The lowercase version has two typographical variants, both inherited from medieval Greek handwriting. One, the most common in modern typography and inherited from medieval minuscule, looks like a reversed "3". The other, also known as lunate or uncial epsilon and inherited from earlier uncial writing, looks like a semicircle crossed by a horizontal bar. While in normal typography these are just alternative font variants, they may have different meanings as mathematical symbols. Computer systems therefore offer distinct encodings for them. In Unicode, the character U+03F5 "Greek lunate epsilon symbol" (ϵ) is provided specifically for the lunate form. In TeX, \epsilon (ϵ) denotes the lunate form, while \varepsilon (ε) denotes the inverted-3 form.

In the times of typewriters, the \epsilon was used for the metarelation \in. You can see yourself how similar they are. I'm finishing reading Halmos' naive set theory and he uses this epsilon as such, and even calls it an epsilon.