Are groups with all the same Hom sets already isomorphic?
Let $A$ be the free group with countably many generators, and let $B$ be group with countably many generators of which one has the relation $g^2=e$.
These are not isomorphic, because a free group is torsion-free and $B$ isn't.
If $C$ is any group with (finite or infinite) cardinality $\lambda$ of which $\kappa$ have order 1 or 2, then we have $$ \begin{align} |\operatorname{Hom}(A,C)| &= \lambda^{\aleph_0} \\ |\operatorname{Hom}(B,C)| &= \lambda^{\aleph_0}\cdot\kappa \end{align} $$ which are equal because $1\le \kappa \le \lambda $.
For the contravariant case, we can dualize Derek's idea: If there are injective homomorphisms $A\to B$ and $B\to A$, then $|\operatorname{Hom}(C,A)|=|\operatorname{Hom}(C,B)|$.
In that case we can take $A$ to be $\mathbb Q^\omega$ and $B=\mathbb Z\times\mathbb Q^\omega$. Then $A$ and $B$ are not isomorphic (because $B$ has an element that is not twice anything), but they easily inject into each other.
(As Derek notes in the comments you can also take $A$ and $B$ to be free groups of different finite ranks $\ge2$, for a finitely generated example. For example, if $A$ is the free group on $\{a,b\}$ and $B$ is the free group on $\{a,b,c\}$, then $A$ injects natively into $B$, and $\{a\mapsto ab, b\mapsto a^2b^2, c\mapsto a^3b^3\}$ generates an injective homomorphism $B\to A$).
If there is a surjective homomorphism $\rho:A \rightarrow B$ then, for any group $C$, $\alpha \mapsto \alpha \circ \rho$ is an injective map ${\rm Hom}(B,C) \to {\rm Hom}(A,C)$.
So, for any pair of groups $A$ and $B$ which are isomorphic to homomorphic images of each other, we have $|{\rm Hom}(A,C)| = |{\rm Hom}(B,C)|$.
There are examples of pairs of non-isomorphic finitely generated groups with this property. An example is given in Theorem 3 of the well-known paper "Some two-generator one-relator non-Hopfian groups" by Baumslag and Solitar
$$A = \langle a,b \mid a^{-1}b^2a=b^3 \rangle,$$ $$B = \langle c,d \mid c^{-1}d^2c=d^3, ([c,d]^2c^{-1})^2=1 \rangle.$$