Taylor's theorem on manifold

One natural way to obtain something like the Taylor expansion of a curve in the manifold is to probe it using a smooth function $\phi:M\to\mathbb{R}$ and expand the composition $\phi\circ f$. To that end, one may write down the Taylor expansion of $\phi\circ f$ in a local chart around $p=f(0)$,

$$\phi(f(t))=\phi(p)+\partial_i\phi \dot f^i(0)\cdot t+\frac12\left((\partial_{ij}\phi)\dot f^i(0)\dot f^j(0)+\partial_i\phi\ddot f^i(0)\right)\cdot t^2+\dots$$

Whereas the first-order term is recognized as $d\phi_{p}(v)$, e.g. as the cotangent vector $d\phi_{p}$ acting on the velocity vector $v=\dot f(0)$, the second-order terms do not have a natural interpretation; neither the second derivative $\partial_{ij}\phi$ nor the 'acceleration' $\ddot f^i(t)$ are tensorial.

If we have a connection $\nabla$, we can rewrite the expansion as

$$\phi(f(t))=\phi(p)+d\phi_{p}(v)+\frac12\left(H\phi^{\nabla}_{p}(v,v)+d\phi_{p}(a)\right)\cdot t^2+\dots,$$

where $H\phi^{\nabla}_{p}$ is the covariant Hessian* of $\phi$ at $p$, which is evaluated on the velocity vector $v$ twice, and $a=\nabla_{\dot f(0)}\dot f$ is the covariant derivative of the velocity vector field along the curve in the direction of $\dot f(0)$ (covariant acceleration).

Both second-order terms are now tensorial, i.e. natural with respect to pullback and pushforward by a smooth morphism $\psi:M\to N$ into a different manifold $N$ with connection $\nabla'$ (i.e. such that $\nabla$ and $\nabla'$ are compatible). Moreover, the parts belonging to the function $\phi$ are neatly separated from the parts belonging to the curve $f$: the former are encoded in covariant tensors while the latter occur as arguments of the covariant tensors, i.e. vectors.

*Note that the Hessian is a bilinear form, and symmetric iff the torsion of $\nabla$ vanishes.

If $M$ is isometrically embedded into some Euclidean space then covariant differentiation (wrt to the induced Levi Civita connection) is nothing but differentiation wrt to an ambient Euclidean space and orthogonal projection onto the tangent space.

Using the Nash Embedding theorem this approach is probably the easiest way in case of (pseudo) Riemannian manifolds (ignoring for a moment the fact that the Nash embedding theorem is a rather deep and nontrivial result). Higher derivatives will be, nevertheless, an unpleasant challenge from the algebraic point of view.