Importance of group action in abstract algebra

To continue along the lines of Ravi’s excellent answer:

Groups are a mathematical representation of symmetry. I’m not exaggerating at all when I say that I consider that sentence to be the most important sentence in group theory. Every time groups arise in mathematics, it’s because the structure you’re considering has some kind of symmetry to it that the group captures. Each element corresponds to a way in which the underlying object is symmetric, and the group as a whole represents all of the symmetries.

Given an object and a symmetry, you can think of a symmetry as a transformation of the object in a particular way that preserves the structure of the object. For example, physically rotating a square by 90 degrees. The action of a group on a set captures the algebraic structure of this transformation, for all the elements of the group. So, the action of a group on a set details precisely how the set transforms under the symmetry described by the group.

Actually, I think the question you should ask is why we deal with abstract groups as sets given some weird looking axioms (which we have become more comfortable with than say with group actions which were the original objects of study!).

Among the first groups that were studied were the permutation groups $S_n$ and you know that these groups are essentially the symmetry groups of a set with $n$ objects. These came up when trying to prove the unsolvability of the quintic and with invariant theory. In addition people were studying symmetries of geometric objects like polygons, polyhedra and the pioneers probably noticed that their symmetries always had the common principles of what we think of as an abstract group.

But the notion of an abstract group did not become a thing until much later in 1854. This wikipedia page does a great job of describing this history which hopefully will shed light on what led mathematicians to come up with group actions/groups: History of Group Theory.

They appear everywhere.

Every module is a special case of a set acted upon by an (abelian) group.

The Erlangen program is a whole system of thought about geometry where you think in terms of groups acting on sets.

A lot of Galois theory works like this, where automorphisms of field extensions work as group actions on the roots of a polynomial.

In general, a group action sort of encapsulates the state of a system when it is transformed with reversible transformations. For example, the configuration of cubes on a Rubik's cube, when acted upon by twisting actions (which constitute a group of reversible transformations.)