How to define a "metric" whose range is not the reals?
Yes, it makes sense, and yes, work has been done on such things. You’ll find work on it under the 2000 AMS Mathematics Subject Classification 46A19, though that does include a few other topics as well.
Non-Archimedean metrics lend themselves to a similar generalization that doesn’t even require a group operation on the linear order in which they take values. Let $\kappa$ be a regular cardinal, and suppose that $d:X\times X\to\kappa+1$ has the following properties:
- For all $x,y\in X$, $d(x,y)=\kappa$ iff $x=y$.
- For all $x,y\in X$, $d(x,y)=d(y,x)$.
- For all $x,y,z\in X$, $d(x,z)\ge\max\{d(x,y),d(y,z)\}$.
For $x\in X$ and $\alpha<\kappa$ let $B(x,\alpha)=\{y\in x:d(x,y)\ge\alpha\}$. Then $\{B(x,\alpha):x\in X\text{ and }\alpha<\kappa\}$ is a base for a topology on $X$. If $\kappa=\omega$, this topology is generated by a non-Archimedean metric $\rho$ on $X$ given by $\rho(x,y)=2^{-d(x,y)}$. A closely related idea from algebra is the notion of a valuation.
There is quite some work done exactly along the lines you suggest. Two sources that answer your question in somewhat different ways are: Flagg's "Quantales and continuity spaces" and Heckmann's "Similarity, Topology and Uniformity".
In more detail, given a quantale (that is a complete lattice together with a binary operation (taken to be commutative, associative, and whose unit is the bottom element of the lattice)) one can define a V space to be a category enriched in V. That amounts to a set $X$ and a function $d:X\times X\to V$ satisfying $d(x,x)\ge \bot$ and $d(x,z)\le d(x,y)+d(y,z)$. In that setting one can do quite a lot of the usual constructions, perhaps most notably the interpretation of Cauchy completeness (see http://ncatlab.org/nlab/show/Cauchy+complete+category). Particular choices of $V$ will recover familiar cases such as: ordinary (non-symmetric) metric spaces, posets, probabilistic metric spaces, as well as Lawvere's fundamental work on generalized metric spaces.
With any V-space one can associate (two) topologies in a way that extends the familiar one from ordinary metric spaces. Varying $V$ yields in this way familiar classes of topologies, most notably the Scott topology. This latter observation explains why V-spaces are studied in Domain Theory.
A nice early result is due to Flagg (and is related to earlier result of Kopperman): Every topological space is V-metrizable if one is allowed to choose $V$ based on the given topology.
A very general answer in the non symmetric case is given by F.W. Lawvere in the paper available for download here entitled "Metric spaces, generalized logic and closed categories".