Can Hom_gp(G,H) fail to be representable for affine algebraic groups?
$\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)$ is not representable.
Let $R$ be a ring of characteristic zero. I claim that $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)(\operatorname{Spec} R)$ is {Nilpotent elements of $R$}. Intuitively, all homs are of the form $x\rightarrow e^{nx}$ with n nilpotent.
More precisely, the schemes underlying $\mathbb{G}_a$ and $\mathbb{G}_m$ are $\operatorname{Spec} R[x]$ and $\operatorname{Spec} R[y, y^{-1}]$ respectively. Any hom of schemes is of the form $y \rightarrow \sum f_i x^i $for some $f_i$ in $R$. The condition that this be a hom of groups says that $\sum f_k (x_1+x_2)^k = (\sum f_i x_1^i)(\sum f_j x_2^j)$. Expanding this, $f_{i+j}/(i+j)! = f_i/i! f_j/j!$. So every hom is of the form $f_i = n^i/i!$, and n must by nilpotent so that the sum will be finite.
Now, let's see that this isn't representable. For any positive integer $k$, let $R_k = C[t]/t^k$. The map $x \rightarrow e^{tx}$ is in $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)(\operatorname{Spec} R_k)$ for every $k$. However, if $R$ is the inverse limit of the $R_k$, there is no corresponding map in $\operatorname{Hom}(\mathbb{G}_a, \mathbb{G}_m)(\operatorname{Spec} R)$. So the functor is not representable.
There is a reasonable salvage, at least if the base scheme is a field: Hom(G,H) is a direct limit of representable subfunctors. See Lemma A.8.13 in the book "pseudo-reductive groups" (where it is used to prove that the scheme-theoretic fixed locus for a linearly reductive group acting on a connected reductive group is always reductive (possibly disconnected) provided the base is a field.