When do automorphisms on open subsets extend

No, this is false. Here is a somewhat sophisticated example; one can probably do simpler.

Let $S\subset \mathbb{P}^3$ be a smooth quartic surface, which contains precisely one line $L$. The Hilbert scheme $X:=\operatorname{Hilb}^2(S) $ is a smooth fourfold, and $\operatorname{Hilb}^2(L) $ embeds naturally as a surface $Y\subset X$; put $U:=X \smallsetminus Y$. Let $Z\in U$ be a length 2 subscheme of $S$; then $Z$ lies on a unique line $\ell_Z≠L$ in $\mathbb{P}^3$, which intersects $S$ along a length 4 subscheme. Associating to $Z$ the residual subscheme of $Z$ in $\ell_Z\cap S$ defines an involution of $U$, which does not extend to $X$.

I think you have an injective homomorphism the "other way around".

As you suspect, Hartog's Lemma implies that $Hom(U,X) = Hom(X,X)$. Now, $Aut(U)$ naturally injects into $Hom(U,X)$. Also, $Aut(X)$ naturally injects into $Hom(U,X)$. The image of $Aut(U)$ in $Hom(U,X)$ lands inside the image of $Aut(X)\to Hom(U,X)$. Thus, in this way, we see that

$Aut(U)$ is a subgroup of $Aut(X)$.

As abx's example shows: this really requires $X$ to be affine. (Otherwise, you just get an injective map $Aut(U) \to Hom(U,X)$.)

The answer to your second question is no. Let $X$ be a three-dimensional algebraic torus, then the automorphism group of $X$ is the extension of $GL_3(\mathbb Z)$ by $X$ (acting by translations. Let $U$ be the complement of three points in general positionin a one-dimensional subtorus. Then the automorphism group of $U$ is the subgroup of $GL_3(\mathbb Z)$ fixing a vector. If there were a surjective homomorphism $Aut(X) \to Aut(U)$, the image of the torus would be a normal abelian subgroup, hence when sent to $GL_2(\mathbb Z)$ it would lie in $\pm 1$, but there is no surjective homomorphism $GL_3(\mathbb Z) \to PGL_2(\mathbb Z)$.