Why is $\operatorname{Hom}(M,N)$ not necessarily an $R$ module?

This is true if $R$ is commutative.

Otherwise, say that you are dealing with left $R$-modules, for instance. If you attempt to define multiplication by $r$ by $(rh)(m) = rh(m)$ for any homomorphism $h \colon M \to N$, then you run into the problem that the mapping $rh$ may not be $R$-linear.

For example, let $h \colon R \to R$ be the identity map. Then $k(s) = (rh)(s)$ would be $rs$. But in general, $k(s1) = rs \neq sr = sk(1)$.

Presumably, you'll want to define $\psi=r.\phi$ by $$\psi(m)=r.\phi(m)$$ (where $r\in R, \phi\in\mathrm{Hom}_R(M,N)$ and $m\in M$.) However, this map, which is a morphism of abelian groups, need not be $R$-linear when $R$ isn't commutative : $R$-linearity would imply that for all $r'\in R$ (and all $m\in M$) $\psi(r'.m)=r'.\psi(m)$, i.e., by $R$-linearity of $\phi$, $$rr'.\phi(m)=r'r.\phi(m)$$ Since $rr'\neq r'r$ in general, there is no reason the above identity should hold.

Any natural definition of $R$-action works only, when $R$ is commutative. For example, if you try to define $$ (rf)(m)=r(f(m)) $$ for all $r\in R$, $f\in Hom_R(M,N)$, $m\in M$, then the mapping $rf$ fails to be homomorphism of $R$-modules in general. If $s\in R$ is such that $sr\neq rs$, then $$ (rf)(sm)=r(f(sm))=r(sf(m))=(rs)(f(m))\neq s((rf)(m)) $$ in general.

OTOH, if one of the modules, $M$ or $N$, is an $(R,R)$-bimodule, then you do get a module structure.