Why does exponentiating the derivative yield the shift operator?

You can just see it as an identity: the shift operator can be expressed in terms of a Taylor series, and then we just compute its closed form.

There are other visualizations for this, though. You can think of $1 + \frac{d}{dx}$ as an infinitesimal shift operator, and exponentiation accumulates all of the infinitesimal shifts up into an actual shift.

In particular, if $E_k$ is the shift-by-$k$ operator, and $\Delta_k = E_k - 1$ then

$$ E_1 = (1 + \Delta_{1/n})^{n} $$

but we know that for small $k$, $\Delta_k f \approx k \frac{df}{dx}$. Thus,

$$ E_1 \approx \left(1 + \frac{1}{n} \frac{d}{dx} \right)^n $$

for large $n$. In fact, both of the following equalities turn out to be true:

$$ E_1 = \lim_{n \to \infty} \left(1 + \frac{1}{n} \frac{d}{dx} \right)^n = e^{d/dx}$$

So you have suggested $e^{t \frac d{dx}} f(x) = f(x+t)$. This is to be expected, because you would formally expect (i) $e^{0 \frac d{dx}}$ to be the identity operator, (ii) $\frac d{dt} [e^{t \frac d{dx}} f(x)] \big|_{t=0} = f'(x)$, and (iii) $e^{t \frac d{dx}} e^{s \frac d{dx}} = e^{(s+t) \frac d{dx}}$. And look, the formula you propose works.

Look here for something more formal: http://en.wikipedia.org/wiki/C0-semigroup