How much of a group $G$ is determined by the category of $G$-sets?

The empty $G$-set is the initial object.

The coproduct of $G$-sets is the disjoint union.

Call a $G$-set “indecomposable” if it is not the coproduct of two non-empty $G$-sets, so an indecomposable $G$-set is just a transitive one.

An epimorphism of $G$-sets is just a surjective map of $G$-sets.

Up to isomorphism, there is a unique transitive $G$-set with an epimorphism to every other transitive $G$-set, namely the regular $G$-set, whose automorphism group is $G$.

So $G$ can be recovered from the category of $G$-sets.

You can also invoke the following theorem :

Two categories have equivalent categories of presheaves if and only if they have equivalent Cauchy completion.

Then remark that a group (seen as a category with one object) is Cauchy complete : its only idempotent is the neutral, which is obviously spit.