What is the definition of "rotation" in a general metric space? (Or a Finsler manifold?)

Mathematical words often get re-used in contexts where there is only an analogy rather than a precise mathematical principle covering precisely all the cases, old and new. This, I think, is what you are encountering with the terminology "rotation". It's a mistake to over-interpret what is going on in with the word "rotation" in each new situation.

Here, for example, is one way that the term "rotation" might grow in use through analogy.

We can certainly define "rotations of the Euclidean plane" with precision. For example:

1. Define $f : \mathbb{R}^2 \to \mathbb{R}^2$ to be a rotation if there exists $\theta \in (0,2\pi)$ and $a,b \in \mathbb{R}$ such that $$f(x,y) = (x \cos \theta + y \sin \theta, - x \sin \theta + y \cos(\theta)) + (a,b) $$

Then we can prove theorems about rotations, for example:

2. $f : \mathbb{R}^2 \to \mathbb{R}^2$ is a rotation if and only if $f$ is an orientation preserving isometry with a unique fixed point.

Now, suppose we are studying the orientation preserving isometries of coordinate Euclidean 3-space $\mathbb{R}^3$. We discover, much to our consternation, that none of them have a unique fixed point.

So, do we shrug our shoulders and say "Euclidean 3-space has no rotations"?

Perhaps, although that feels uncomfortable given our real-world experience with 3-dimensional rotations.

There is, however, another possibility, namely to change our analogy by proving that 1 and 2 are equivalent to

3. $f : \mathbb{R}^2 \to \mathbb{R}^2$ is a rotation if and only if it is an orientation preserving isometry and its fixed point set is a codimension 2 affine subspace.

And now, perhaps, we are happy, because this indeed generalizes very nicely to Euclidean spaces of all dimensions. And perhaps now we think know what rotations are.

Well, maybe. Every time we go down roads of further and further generalization, we may be tempted to stretch the analogy further and make some grand generalized "rotation" definition. But somehow that misses the point.

The real point is: We should study isometries of $\mathbb{R}^2$, or $\mathbb{R}^3$, or $\mathbb{R}^n$, or CAT(0) Riemannian metric spaces, or whatever new context we encounter, for what they really are. If analogy with rotations of $\mathbb{R}^2$ help us understand isometries in these new contexts, by all means reuse the terminology. If it is misleading, or if it just is unhelpful, then don't use the terminology. At the very least, don't stretch the analogy beyond the breaking point.

There is no widely used definition of rotation in a general metric space. This becomes clear after Googling a bit and reading the comments and answers of this question and your previous question. Therefore I would say there is no such thing as the definition.

You can define anything you like as long as it makes some sense. Your definition of $G_x$ definitely makes sense, since when $X = \mathbb{R}^n$ and $d$ is the Euclidean norm, the group $G_x$ is equal to the orthogonal group $O(n)$.

Generalizing proper rotations

The group $G_x$ generalizes both proper and improper rotations. The word rotation sometimes includes improper rotations and sometimes it does not. I have thought about how to generalize proper rotations to arbitrary metric spaces.

What's the difference between proper and improper rotations? I would say that a proper rotation is continuous, in the sense that you can move the elements of the metric space by a lot of small bits in order to obtain the complete rotation, while preserving the distance at all times.

You could use $$ \begin{array}{} P_x &=& \{f \in G_x \ |\ \exists (p \in [0,1] \to G_x): p(0) = \text{id}_X \land p(1) = f\ \land \\ && \qquad \forall(\alpha_1 \in [0, 1], y \in X, \epsilon > 0): \exists(\delta > 0): \forall(\alpha_2 \in [0,1]): \\ && \qquad \qquad |\alpha_1 - \alpha_2| < \delta \to d(p(\alpha_1)(y), p(\alpha_2)(y)) < \epsilon \\ &&\}, \end{array} $$ which is a subgroup of $G_x$, to generalize proper rotations to general metric spaces. For $\mathbb{R}^n$ with the Euclidean metric, the group is equal to the special orthogonal group $SO(n)$.

For $\mathbb{R}^n$ with the taxicab metric, $P_x$ will be the trivial group, since no continuous rotations are possible with the taxicab metric, only rotations of 90 degrees.