What's so "natural" about the base of natural logarithms?

Differentiation and integration is precisely why it is considered natural, but not just because $$\displaystyle\int \frac{1}{x} dx=\ln x$$

$e^x$ has the two following nice properties

$$ \frac{d}{dx} e^x=e^x $$

$$ \int e^x dx=e^x+c $$

If we looked at $a^x$ instead, we would get:

$$\frac {d} {dx} a^x= \frac{d}{dx} e^{x\ln(a)}=\ln(a) \cdot a^x$$

$$\int a^x dx= \int e^{x\ln(a)} dx=\frac{a^x}{\ln(a)}+c$$

So $e$ is vital to the integration and differentiation of exponentials.

If you know some linear algebra, then here is an abstract reason: $e^x$ is the unique eigenvector of eigenvalue $1$ of the derivative $D$ acting on, say, the space of smooth functions on $\mathbb{R}$. Why is this important? The study of solutions of linear differential equations with constant coefficients is equivalent to the study of nullspaces of operators which are polynomials in $D$, e.g. operators of the form $\sum a_k D^k$. Any such operator automatically commutes with $D$, so this nullspace splits up into eigenspaces of $D$. That's why solutions to linear differential equations with constant coefficients can be expressed as sums of complex exponentials. The choice of $e$ makes it particularly easy to see what the eigenvalue is: the eigenvalue of the eigenvector $e^{\lambda x}$ is $\lambda$.

The wikipedia article on e tells a bit of the story.

One example is an account that starts with 1.00 and pays 100% interest per year. If the interest is credited once, at the end of the year, the value is 2.00; but if the interest is computed and added twice in the year, the 1 is multiplied by 1.5 twice, yielding 1.00×1.5² = $2.25. Compounding quarterly yields 1.00×1.25⁴ = 2.4414…, and compounding monthly yields 1.00×(1.0833…)¹² = 2.613035….

Bernoulli noticed that this sequence approaches a limit (the force of interest) for more and smaller compounding intervals. Compounding weekly yields 2.692597…, while compounding daily yields 2.714567…, just two cents more. Using n as the number of compounding intervals, with interest of 100%/n in each interval, the limit for large n is the number that came to be known as e; with continuous compounding, the account value will reach 2.7182818…. More generally, an account that starts at $1, and yields (1+R) dollars at simple interest, will yield e^R dollars with continuous compounding.

Additionally, it is the base of the exponential function y = k^x, finding a specific value for k where d/dx k^x = k^x. That is, the rate of change of the exponential curve at any point is equal to the y value of the curve at that point.