Set of homomorphisms form a group?
The set of all homomorphisms between two groups naturally forms a groupoid rather than a group; the objects of the groupoid are the homomorphisms and the morphisms are given by pointwise conjugation, so if $\varphi_1, \varphi_2 : G \to H$ are two homomorphisms then a morphism between them is an element $h \in H$ such that $\varphi_1(g) = h \varphi_2(g) h^{-1}$ for all $g \in G$.
This is a special case of the construction of functor categories, thinking of groups as one-object categories.
Some other relevant keywords from category theory: enriched category, preadditive category (what I would prefer to call an $\text{Ab}$-enriched category). The category $\text{Vect}$ of vector spaces is enriched over itself, as is the category $\text{Ab}$ of abelian groups, the category $\text{Cat}$ of (small) categories, and its subcategory $\text{Gpd}$ of groupoids.
I realize I've dodged the original question to some extent. There's a natural extra condition to ask for when you put additional structure on homsets, namely composition ought to respect that structure. So if you want to enrich the category of groups over itself, then it would be nice if composition
$$\text{Hom}(G, H) \times \text{Hom}(H, K) \to \text{Hom}(G, K)$$
were a group homomorphism. But this implies that $\text{Hom}(G, G)$ comes equipped with two monoid structures, one given by the enrichment and one given by composition, which satisfy the compatibility relation from the Eckmann-Hilton argument. It then follows that the monoid structures must be isomorphic and commutative, but now it follows that every group homomorphism $G \to G$ is an isomorphism and also that $\text{Aut}(G)$ is always commutative, both of which are clearly a contradiction.
(So why can we enrich the category of abelian groups over itself? The reason is that we give $\text{Ab}$ a different monoidal structure, namely the tensor product, which changes the condition we want on composition to
$$\text{Hom}(A, B) \otimes \text{Hom}(B, C) \to \text{Hom}(A, C)$$
being a group homomorphism, which can happen. Now the compatibility relation on $\text{Hom}(A, A)$ coming from its two monoid structures turns it into a ring. But the category of groups, unlike the category of abelian groups, doesn't have a natural notion of tensor product.)