Differentiability in metric spaces

Riemannian manifolds are the primary example of metric spaces which have a useful notion of differentiability, despite not necessarily being $\mathbb R^n$ or any kind of vector space. (One does not actually need the Riemannian structure to define derivatives; only the smooth structure is needed. But you asked about metric spaces, and general smooth manifolds do not have a preferred metric.)

One can push a bit of calculus into the realm of metric spaces that are not smooth manifolds. This subject (Analysis on metric spaces) gets pretty technical and axiomatic quickly, but there are strong recent results, with applications to other areas like embeddings of infinite graphs into normed spaces. A standard reference is the survey Nonsmooth calculus by Heinonen:

We survey recent advances in analysis and geometry, where first order differential analysis has been extended beyond its classical smooth settings. Such studies have applications to geometric rigidity questions, but are also of intrinsic interest. The transition from smooth spaces to singular spaces where calculus is possible parallels the classical development from smooth functions to functions with weak or generalized derivatives. Moreover, there is a new way of looking at the classical geometric theory of Sobolev functions that is useful in more general contexts.

Metric Differentiability was introduced first in the paper

B. Kirchheim, Rectifiable metric spaces, local structure and regularity of the Hausdorff measure. Proc. AMS, 121 (1994), 113-123

It is defined for maps from (subsets of) Euclidean spaces to general metric spaces: $f: \mathbb{R}^n \supset A \to (X,d)$.

The "derivative of $f$ at point $x$ in the direction $v \in \mathbb{R}^n$" is $$ \liminf_{t \searrow 0} \frac{d(f(x+tv),f(x))}{t} \ .$$ Notice that no linear structure is required in the target space for this to make sense.

The big idea is that for Lipschitz maps, at a.e. $x \in A$, this quantity will be a seminorm (in $v$). This is analogue of the more famous Radamacher's theorem. See how simple the assumptions are!

Kirchheim used this seminorm to define a Jacobian with which he proved an area formula in this context: $\mathcal{H}^n$-measure of a the image of a subset equals the integral of this Jacobian over the set.