Intuitive explanation for why $\left(1-\frac1n\right)^n \to \frac1e$
The point is that $1-\frac{1}{n}$ is less than $1$, so raising it to a large power will make it even less-er than $1$. On the other hand, $1+\frac{1}{n}$ is bigger than $1$, so raising it to a large power will make it even bigger than $1$.
There's been some brouhaha in the comments about this answer. I should probably add that $(1-\epsilon(n))^n$ could go to any value less than or equal to $1$, and in particular it could go to $1$, as $n$ increases. It so happens that in this example, it goes to something less than $1$. The reason it goes to something less than $1$ is because we end up raising something sufficiently less than $1$ to a sufficiently high power.
Perhaps think about the binomial expansions of $\left(1 + \frac{1}{n}\right)^n$ and $\left(1 - \frac{1}{n}\right)^n$. The first two terms are $1 + n \frac{1}{n}$ and $1 - n \frac{1}{n}$ respectively. And after that the terms in $\left(1 + \frac{1}{n}\right)^n$ are all positive, whereas the terms in $\left(1 - \frac{1}{n}\right)^n$ alternate. So the difference between the two limits is going to be at least 2.
The true issue is not why changing the sign has such an impact, it is why adding such a small quantity as $\dfrac1n$ drastically changes the result.
$$1^n\to1\text{ vs. }\left(1+\frac1n\right)^n\to e$$
(and very similarly $\left(1-\frac1n\right)^n\to e^{-1}$.)
The reason is that the tiny quantity gets multiplied over and over so that it becomes a finite quantity,
$$\left(1+\frac1n\right)\left(1+\frac1n\right)\left(1+\frac1n\right)\cdots=1+\frac1n+\frac1n+\frac1n+\cdots>2$$ as there are $n$ terms $\dfrac1n$ (and yet others). The "tininess" of the terms is well compensated by the amount of terms.
Also notice that the "asymmetry" shown by $e-1\ne 1-e^{-1}$ is just due to the non-linearity of the exponential.