"Alternatives" to Natural Transformations

There's nothing preventing you from taking any class, and then making any category $\mathcal{C}$ you want for which $\mathrm{ob}(\mathcal{C})$ is that class, as long as $\mathcal{C}$ satisfies the axioms for a category. If you want the class to be the class of all functors $\mathcal{A}\to\mathcal{B}$ for some other categories $\mathcal{A}$ and $\mathcal{B}$, that's fine.

Analagously, you could take abelian groups $A$ and $B$, form the set of all group homomorphisms $A\to B$, and then make that set into an abelian group $C$ in any way you want.

Of course, the usefulness of doing these things in any way but the usual way is a separate issue.

If you have some experience with algebraic topology, here's some motivation as to why natural transformations are a useful notion of morphism between functors. One can rephrase the definition of natural transformation as follows.

Let $I$ be the category with two objects, $0$ and $1$, and one nontrivial morphism $0 \to 1$. For any category $\mathcal{C}$, we have two "inclusion" functors $i_0, i_1 : \mathcal{C} \to \mathcal{C} \times I$. Then a natural transformation between functors $F, G: \mathcal{C} \to \mathcal{D}$ gives exactly the same data as a functor $H : \mathcal{C} \times I \to \mathcal{D}$ such that, the compositions $H \circ i_0$ and $H\circ i_1: \mathcal{C} \to \mathcal{C} \times I \to \mathcal{D}$ are $F$ and $G$, respectively.

This is exactly analogous to the definition of a homotopy between continuous maps, i.e. a morphism between morphisms. In fact, higher category theory draws a lot of inspiration from algebraic topology.

Hope this helps!

Here is, in my opinion, a good explanation of why natural transformations are the most natural notion of morphism between functors, based on the algebra of $\mathbf{Cat}$.

Suppose $\mathcal{C}$ is a category.

The objects of $\mathcal{C}$ are precisely the functors $\mathbf{1} \to \mathcal{C}$, where $\mathbf{1}$ is the terminal category. (one object, no nonidentity arrows)

The arrows of $\mathcal{C}$ are precisely the functors $\mathbf{2} \to \mathcal{C}$, whre $\mathbf{2}$ is the arrow category. (two objects, one nonidentity arrow from one to the other)

Now, suppose there was a good notion of a functor category: a category $\mathcal{D}^\mathcal{C}$ whose objects are functors $\mathcal{C} \to \mathcal{D}$. If we insist the usual relationship bewteen products and exponentials holds, then the following notions are equivalent:

• A functor $\mathbf{2} \to \mathcal{D}^\mathcal{C}$
• A functor $\mathbf{2} \times \mathcal{C} \to \mathcal{D}$

Objects of the first type tell us what morphisms between functors should be. Objects of the second type we can handle explicitly, and it's not hard to show they give the usual definition of natural transformation.