Is there any geometric way to characterize $e$?

For a certain definition of "geometrically," the answer is that this is an open problem. You can construct $\pi$ geometrically in terms of the circumference of the unit circle. This is a certain integral of a "nice" function over a "nice" domain; formalizing this idea leads to the notion of a period in algebraic geometry. $\pi$, as well as any algebraic number, is a period.

It is an open problem whether $e$ is a period. According to Wikipedia, the answer is expected to be no.

In general, for a reasonable definition of "geometrically" you should only be able to construct computable numbers, of which there are countably many. Since the reals are uncountable, most real numbers cannot be constructed "geometrically."

The area beneath the reciprocal function $$\frac{1}{x}$$ from $x=1$ to $x=e$ is $1$. Though this isn't really geometric like you want, it is still a clear way to see $e$ physically.

Debeaune asked Descartes this problem in a letter in 1638:

Consider a curve $y=f(x)$. Lets consider the tangent line $t(x)$ through the point $(x_{0},y_{0})$ which would look like $t(x) = y_{0} + f'(x_{0})\cdot(x-x_{0})$. What curve has the property that, every such tangent line intersects the $x$ axis at $x_{0}-1$, i.e., $$ t(x_{0}-1)=0 $$ What curve can do this? Only $y=C\exp(x)$...where $C$ is some nonzero constant.

For a thorough derivation, see http://pqnelson.wordpress.com/2012/06/03/exponential-function/