What does the letter epsilon signify in mathematics?

The greek letter epsilon, written $\epsilon$ or $\varepsilon$, is just another variable, like $x$, $n$ or $T$.

Conventionally it's used to denote a small quantity, like an error, or perhaps a term which will be taken to zero in some limit.

It's possible that you are confusing it with the set membership symbol $\in$, which is something different. When you see $x\in X$ it means that $X$ is a set, and $x$ is a member of the set. For example,

$$1\in \{1,2,3\}$$

is true, but

$$4\in\{1,2,3\}$$

is false.

Hilbert's epsilon-calculus used the letter $\varepsilon$ to denote a value satisfying a predicate. If $\phi(x)$ is any property, then $\varepsilon x. \phi(x)$ is a term $t$ such that $\phi(t)$ is true, if such $t$ exists. One can define the usual existential and universal quantifiers $\exists$ and $\forall$ in terms of the $\varepsilon$ quantifier:

$$\begin{eqnarray} \def\hil#1{#1(\varepsilon x. #1(x))} \exists x.\phi(x) & \equiv & \hil{\phi}\\ \forall x.\phi(x) & \equiv & \phi(\varepsilon x.\lnot\phi(x)) \end{eqnarray} $$

Here's a not too well-known instance of the use of $\varepsilon$ in mathematics:

One somewhat well-known transformation for accelerating the convergence of a sequence is the Shanks transformation (after Daniel Shanks, who is probably more well-known for his number-theoretic contributions). What the Shanks transformation essentially does, assuming that the sequence given is a sequence of Taylor polynomials evaluated at a certain argument, is to transform this sequence of Taylor approximants into a sequence of Padé rational approximants.

The Shanks transformation of a sequence can be expressed as a ratio of two determinants, but there is a more efficient realization of this, the Wynn $\varepsilon$ algorithm:

$$\varepsilon_{k+1}^{(n)}=\varepsilon_{k-1}^{(n+1)}+\frac1{\varepsilon_{k}^{(n+1)}-\varepsilon_k^{(n)}}$$

where $\varepsilon_0^{(n)}=S_n$ is the sequence to be transformed.