Dynamical system defined with a non-abelian group

What do you mean by "modeled"? If you want to keep the idea of time, I think there is little "modeling" in that sense for non-abelian groups (I'll stand corrected if some example arises). If by "model" you mean applications to physics, the notion of an action of a group on a given manifold, for example, has intrinsic relations and usefulness to the notion of symmetry (which is obviously invaluable in physics, and where motivated non-abelian groups arise naturally - $\mathrm{SO}(3)$, for example). In this sense, the dynamical system can be interpreted as there being a way to move around the space via certain specific movements which are of interest to the specific case and which behave in "concatenable" ways. Implicitly, this is also what it is really happening when you are given an action of $\mathbb{R}$ or $\mathbb{Z}$: the way of moving being flowing forwards/backwards on time.

If you want to know the usefulness of the notion (in mathematics itself, say) in perhaps surprising situations, then there are infinite. For example, an elementary usefulness is knowing that $\mathrm{SO}(n)$ (non-abelian) acts on the sphere $S^{n-1}$ with isotropy "equal to" $\mathrm{SO}(n-1)$, and hence $\mathrm{SO}(n)/\mathrm{SO}(n-1) \simeq S^{n-1}$. This allows one to conclude rather cleanly, for instance, that $\mathrm{SO}(n)$ is path-connected for every $n$ (there is a result which says that if $G/H$ is connected and $H$ is connected then $G$ is connected).

More interestingly, one can give a counter-example to the "pseudo-Poincaré conjecture" that a compact manifold with the homology of a sphere is homeomorphic to the sphere. This is done by using the fact that the quotient of a properly discontinuous action of a group $G $ on a simply-connected manifold $M$ has fundamental group equal to $G$ itself, which is a good application by itself (for example, $\mathbb{Z}\oplus \mathbb{Z}$ acts on $\mathbb{R}^2$ by translation, with quotient being the torus, which allows us to conclude that $\pi_1(T^2)=\mathbb{Z}\oplus \mathbb{Z}$. Another example is the action of $\mathbb{Z}/2\mathbb{Z}$ on the sphere which furnishes the projective space). Omitting details, one constructs a properly discontinuous action of a perfect $G$ group on $S^3$. By the theorem I mentioned, the fundamental group of the space $X$ given by the quotient of the action is necessarily $G$. In particular, it can't be homeomorphic to $S^3$. However, by Hurewicz, $H_1(X)=0$, and by duality $H_2(X)=0$ (since $H^1(X)=0$ by the universal coefficient's theorem).

With $G$ higher-dimensional you don't want to think of its action as representing translations in time, but rather translations in space.