Exterior Derivative vs. Covariant Derivative vs. Lie Derivative

Short answer:

• the exterior derivative acts on differential forms;
• the Lie derivative acts on any tensors and some other geometric objects (they have to be natural, e.g. a connection, see the paper of P. Petersen below);
• both the exterior and the Lie derivatives don't require any additional geometric structure: they rely on the differential structure of the manifold;
• the covariant derivative needs a choice of connection which sometimes (e.g. in a presence of a semi-Riemannian metric) can be made canonically;
• there are relationships between these derivatives.

For a longer answer I would suggest the following selection of papers

1. T. J. Willmore, The definition of Lie derivative
2. R. Palais, A definition of the exterior derivative in terms of Lie derivatives
3. P. Petersen, The Ricci and Bianchi Identities

Of course, there is a lot more to say.

---

Edit. I decided to extend my answer as I believe that there are some essential points which have not been discussed yet.

1. An encyclopedic reference that treats all these derivatives concurrently at a modern level of generality is
   I.Kolar, P.W. Michor, J. Slovak, Natural Operations in Differential Geometry (Springer 1993), freely available online here.
   I would not even dare to summarize this resource since it has an abysmal deepness and all-round completeness, and indeed covers all the parts of the original question.
   Moreover, I believe that the bibliography list of this book contains almost any further relevant reference.
2. As it has been already mentioned by many in this discussion, these operations are intimately related. It cannot be overemphasized that the most important feature that they all share is naturality (they commute with pullback, and this, in particular, makes them coordinate-free).
   See KMS cited above and its bibliography, and specifically the following references may be useful:
   R. Palais, Natural Operations on Differential Forms, e.g. here or here.
   C.L. Terng, Natural Vector Bundles and Natural Differential Operators, e.g. here
3. It turns out that their naturality forces them to be unique if we impose on them some basic properties, such as $d \circ d = 0$ for the exterior derivative. One way to prove that and further references could be found in:
   D. Krupka, V. Mikolasova, On the uniqueness of some differential invariants: $d$, $[,]$, $\nabla $, see here.
   Also it is interesting that the Bianchi identities for the connection follow from the naturality and the property $d \circ d = 0$ for the exterior derivative, see
   Ph. Delanoe, On Bianchi identities, e.g. here.
4. The reference list that I produce here is too far from being complete in any sense. I only would add one classical treatment that I personally used to comprehend some of the fundamental notions related to Lie derivatives (in particular, the Lie derivative of a connection!):
   K. Yano, The Theory Of Lie Derivatives And Its Applications, freely available here

Indeed, my comments are speculative and sparse. I wish if this question were answered by someone like P. Michor, to be honest :-)

Since I don't have the time to give a super-detailed answer, allow me to just summarize some things that others have said, adding some additional points in the process. Hopefully this will be at least somewhat helpful.

Basic differences:

• The exterior derivative and Lie derivative are defined in terms of the structure of a smooth manifold. By contrast, the choice of a connection is an additional structure.

• All three agree on smooth functions. However, they generalize differently:

• The exterior derivative takes differential forms as inputs.

• Connections take sections of a vector bundle (such as tensor fields) as inputs, and differentiation is done with respect to a vector field.

• The Lie derivative takes tensor fields as inputs, and differentiation is done with respect to a vector field.

Exterior derivative: The main feature here is $d^2 = 0$.

To me, the exterior derivative is the differentation operator we need for Stokes' Theorem to make sense. The $d^2 = 0$ property is dual to saying that "the boundary of a boundary is empty," and is the very thing that makes de Rham cohomology work. I sometimes think about $d^2 = 0$ as describing the commutativity of second derivatives.

Connections: The main feature here is differentiation along curves.

A choice of connection allows us to define the derivative of a vector field (more generally, a section of a vector bundle) with respect to another vector field. From there, we can define the notion of a "covariant derivative" along curves.

Connections generalize the case of $\mathbb{R}^n$, where

$$\nabla_XY := X^i\frac{\partial Y}{\partial x^i}$$

Connections also let us define the concepts of "parallel transport" and "torsion." When a Riemannian metric is given, there is a canonical choice of connection (the Levi-Civita connection) which gives lots of geometric information. In particular, many classical formulas from the differential geometry of curves and surfaces can be phrased in terms of connections.

Lie derivative: The main feature here is the relationship with integral curves and flows, and the fact that $\mathscr{L}_XY = XY - YX$.

Like connections, the Lie derivative also defines a derivative of a vector field (more generally, tensor fields) with respect to another vector field. Intuitively, the Lie derivative $\mathscr{L}_XY$ is the instantaneous change of $Y$ along the integral curves defined by $X$. This intuition comes directly from the definition:

$$\mathscr{L}_XY|_p := \lim_{t \to 0}\frac{D\phi_{-t}(Y_{\phi_t(p)}) - Y_p}{t},$$ where $\phi$ is the flow of $X$.

However, unlike connections, Lie derivatives do not give a well-defined directional derivative of vector fields "along curves." The following problem from Lee's Riemannian Manifolds book illustrates this:

Problem 4-3: b) There exists a vector fields $V,W$ on $\mathbb{R}^2$ such that $V=W=\partial_1$ on the $x^1$-axis, but the Lie derivatives $\mathcal L_V(\partial_2)$ and $\mathcal L_W(\partial_2)$ are not equal on the $x^1$-axis.

Pretty much all of this can be found in Lee's "Smooth Manifolds" and "Riemannian Manifolds" books.

Let me focus on the difference between Lie derivatives and covariant derivatives. Suppose I have a manifold with a connection $\nabla$ and a point $p$ in the manifold. Let $v$ be a vector field on $M$ and take $\xi \in T_pM$. The point to stress is that $\xi$ is not a vector field (although in practice it is often a vector field evaluated at $p$). We can then obtain $\nabla_{\xi}v \in T_pM$. Thus, covariant derivatives let you take directional derivatives of vector fields.

Lie derivatives of vector fields cannot quite be interpreted this way. The symbol "$\mathcal{L}_{\xi} v$" is not defined because $\xi$ is not a vector field. However, if we take a vector field $X$, we can think of $\mathcal{L}_{X} v$ (which is a new vector field) as the derivative of $v$ as we walk along the integral curves of $X$.

To understand the difference between these two ideas, consider working in coordinates. The Lie derivative will take the form $(\mathcal{L}_{X} v)^a=X^b \partial_b v^a-v^b \partial_b X^a$. The second term indicates that the first derivative of $X^\mu$ is involved when differentiating along flow lines. There is no sensible way to differentiate $v$ in the direction of a vector unless the vector is a vector field or you have additional structure that tells you how to connect nearby tangent spaces (a connection). (The only obvious guess would be $X^b \partial_b v^a$ which gives different vectors in different coordinate systems.)