Can Isomorphism between categories be defined only if both categories are small?

No. But the answer in detail depends on the foundations you are using. For instance, in $\mathsf{ZFC}$ we may interpret classes using formulas (up to equivalence). Then, a category $C$ has a formula $\mathrm{Ob}(C)$ with one variable such that "$x$ is an object of $C$" means that $\mathrm{Ob}(C)(x)$ holds. Also, we have formulas $\mathrm{Mor}(C)$, $\mathrm{dom}(C)$, $\mathrm{cod}(C)$, $\mathrm{Comp}(C)$, $\mathrm{Id}(C)$ such that the category axioms are satisfied. Now, a functor $F : C \to D$ consists of two formulas $F_O$ and $F_M$ such that

• $\forall x ( \mathrm{Ob}(C)(x) \Rightarrow \exists ! y (\mathrm{Ob}(D)(y) \wedge F_O(x,y))$

i.e., every object of $C$ is mapped to some specified object of $D$; one usually writes $F(x)=y$ instead of $F_O(x,y)$. Likewise for morphisms:

• $\forall f (\mathrm{Mor}(C)(f) \Rightarrow \exists ! g ( \mathrm{Mor}(D)(g) \wedge F_M(f,g))$

• $\forall f,g,x,x',y,y' (F_M(f,g) \wedge \mathrm{dom}(C)(f,x) \wedge \mathrm{cod}(C)(f,x') \wedge F_O(x,y) \wedge F_O(x',y') \Rightarrow \mathrm{dom}(D)(g,y) \wedge \mathrm{cod}(D)(g,y'))$

i.e., if $f$ is a morphism in $C$ from $x$ to $x'$, then $F(f)$ is a morphism in $D$ from $F(x)$ to $F(x')$,

• compabilities with respect to identities and composition (which I won't write down here).

The composition of two functors $F : C \to D$, $G : D \to E$ is the functor $G \circ F : C \to E$ defined by $(G \circ F)_O(x,z) :\Leftrightarrow \exists y (F_O(x,y) \wedge G_O(y,z))$, likewise for $(G \circ F)_M$.

Now, $F$ is an isomorphism iff $F_O$ and $F_M$ are bijective, i.e.

• $\forall y ( \mathrm{Ob}(D)(y) \Rightarrow \exists ! x (\mathrm{Ob}(C)(x) \wedge F_O(x,y))$
• $\forall g (\mathrm{Mor}(D)(g) \Rightarrow \exists ! f ( \mathrm{Mor}(C)(f) \wedge F_M(f,g))$

In this case, the inverse functor $F^{-1}$ is defined by $F^{-1}_O(y,x) :\Leftrightarrow F_O(x,y)$ and $F^{-1}_M(g,f) :\Leftrightarrow F_M(f,g)$.

Bijections can also be defined on proper classes.
So it doesn't matter if the category is (locally) small.