Coend computation continued
I agree with Reid's answer, but I want to add a bit more.
Putting Reid's calculation into a more general setting, if $A$ is any category then $$ \int^{a \in A} \mathrm{hom}_A (a, a) = (\mathrm{endomorphisms\ in\ } A)/\sim $$ where $\sim$ is the (rather nontrivial) equivalence relation generated by $gh \sim hg$ whenever $g$ and $h$ are arrows for which these composites are defined.
You can see confirmation there that your instinct about traces was right. If we wanted to define a 'trace map' on the endomorphisms in $A$, it should presumably satisfy $\mathrm{tr}(gh) = \mathrm{tr}(hg)$, i.e. it should factor through $\int^a \mathrm{hom}(a, a)$.
In fact, Simon Willerton has done work on 2-traces in which exactly this coend appears. See for instance these slides, especially the last one.
You can see in that slide something about the dual formula, the end $$ \int_{a \in A} \mathrm{hom}_A(a, a). $$ By the "fundamental fact" I mentioned before, this is the set $$ {}[A, A](\mathrm{id}, \mathrm{id}) = \mathrm{Nat}(\mathrm{id}, \mathrm{id}) $$ of natural transformations from the identity functor on $A$ to itself. Evidently this is a monoid, and it's known as the centre of $A$. For example, when $G$ is a group construed as a one-object category, it's the centre in the sense of group theory. So your set might, I suppose, be called the co-centre of $A$.
The step $\int_{a \in A} \mathrm{Set}(\mathrm{hom}_A(a, a), S) = \mathrm{Nat}(\mathrm{hom}_A(-,-),S)$ does not really make sense, because $a \mapsto \mathrm{hom}_A(a,a)$ is not a functor. And $\int^{a \in A} \mathrm{hom}_A(a,a)$ is not equal to $\mathrm{Ob}(A)$ in general. For instance, let $G$ be a group and let $A = BG$ be the groupoid with a single object with automorphism group $G$. Then $\int^{a \in A} \mathrm{hom}_A(a,a)$ can be identified with the abelianization set of conjugacy classes of $G$. In general, $\int^{a \in A} \mathrm{hom}_A(a,a)$ is the "Hochschild homology" of the category $A$.
Edit: I was originally thinking of the case of $G$ abelian, and generalized wrongly to the nonabelian case. As atonement, let me write out the computation directly from the definition of coend:
$$\int^{a \in A} \mathrm{hom}_A(a, a) = \operatorname{colim} \left[ \coprod_{f:a \to b} \mathrm{hom}_A(b,a)\rightrightarrows \coprod_{a\in A} \mathrm{hom}_A(a,a)\right] = \operatorname{colim} [G \times G \rightrightarrows G]$$
where the two maps send $(g, h)$ to $gh$ and $hg$ respectively. So the coend is the quotient of $G$ in the category of sets by the relation $gh = hg$, or $g = hgh^{-1}$, i.e., it is the set of conjugacy classes of elements of $G$.