A general definition of symmetry in mathematics

If an abstract category-theoretic answer will satisfy you, then you could say something very general like

A symmetry of a morphism $\phi:A\to B$ means a pair $(\alpha,\beta)$ of automorphisms of $A$ and $B$ respectively, such that $\beta \circ \phi = \phi \circ \alpha$.

One easily sees that the symmetries of any $\phi$ constitute a group. If $\phi$ is mono (or epi) then $\beta$ determines $\alpha$ (or vice versa), and the symmetry group is a subgroup of $\operatorname{Aut}(B)$ (or $\operatorname{Aut}(A)$).

As a special case, when $\phi$ is an identity morphism, a symmetry is just an automorphism.

Ordinary geometric symmetries arise in this framework in the category of metric spaces and distance-preserving maps, where $\phi$ is the inclusion map from $A\subseteq \mathbb R^n$ into $\mathbb R^n$.