How to prove Cayley's theorem using Category Theory

There nothing wrong with looking at elements of the group. If a group is seen as a one-object category, then what would be thought of as elements in traditional abstract algebra now become identified as the morphisms of the category. And one is certainly “allowed” to deal with elements of morphism sets.

Let me sum up the discussion in the comments.

This proof that is sketched isn't particularly categorical but it does relate to the Yoneda embedding $\mathcal{C} \to \mathsf{Set}^{\mathcal C^{\rm op}}$ where we map objects $A$ of $\mathcal{C}$ to their hom-functors $h^{A} : \mathcal{C}^{\rm op} \to \mathsf{Set}$. My guess is that the authors intend to explain the categorical connection later.

In the case were $\mathcal{C}$ is a group with a single object $A$, then $\mathcal{C} = \mathcal{C}^{\rm op}$ and $h^A : \mathcal{C} \to \mathsf{Set}$ takes $A$ to $\operatorname{Hom}(A,A) = G$ and group elements $g : A \to A$ to $\mathsf{Set}$-morphisms $\bar g(h) = g \circ h$. So this gives a map from $G = \operatorname{Hom}(A,A) \to \operatorname{Perm}(G)$.

The subgroup of $\operatorname{Perm}(G)$ which is the image of $\operatorname{Hom}(A,A)$ is the group of natural transformations of $h^A$. A natural transformation of $h^A$ looks like

$$\require{AMScd} \begin{CD} A @>g>> A\\ @V h^A V V @VV h^A V\\ \operatorname{Hom}(A,A) @>>\bar g> \operatorname{Hom}(A,A) \end{CD} $$

You can see this all on Wikipedia. Hopefully the above gives you some categorical context for Cayley's thoerem.