Automorphism group of a scheme

The answer is yes when the scheme is flat and projective over the base. This follows from the existence of the Hom scheme, which in turn is proven via the existence of the Hilbert scheme.

A readable reference is Nitin Nitsure's part of the book Fundamental algebraic geometry. In particular Theorem 5.23 in his notes states that when S is noetherian, X projective and flat over S, and Y quasi-projective over S, then there exists a Hom-scheme parametrizing morphisms from X to Y over S. (The precise definition of the functor that this scheme represents is given in the text). The automorphism group scheme of X is then an open subscheme of $\mathrm{Hom}_S(X,X)$.

Addendum: The proof that Isom is open in Hom is very similar to the proof that Hom is open in Hilb. The map from the Hom scheme to $\mathrm{Hilb}_{X \times_S Y / S}$ is given by associating to a morphism $f : X \to Y$ its graph. Now the image consists of those $Z \subset X \times_S Y$ such that projection onto X induces an isomorphism $Z \cong X$, and the crucial part of the proof is showing that this is an open condition. (In Nitsure's notes this is Theorem 5.22.(b).) But then the condition that Z maps isomorphically onto Y is also open, and this is exactly the condition that defines the Isom scheme inside the Hom scheme.

The automorphism group functor is not representable in general. Consider for instance the case of an algebraically closed field $k$ and $\mathbf A^2=\mathrm{Spec}\ k[x,y]$. Assume the automorphism group functor for this scheme is representable by an algebraic space $Aut$. For each $n$ we have an automorphism $(x,y)\mapsto(x+\sum_{1\leq i\leq n}t^iy^i,y)$ over $k[t]/(t^{n+1})$ giving a set of compatible morphisms $\mathrm{Spec}k[t]/(t^{n+1})\to Aut$. That system comes from a morphism $\mathrm{Spec}\ k[[t]]\to Aut$ (see Emerton's response to this question, which works also for maps into algebraic spaces at least as $k$ is algebraically closed). However, the corresponding automorphism would have to be $(x,y)\mapsto(x+\sum_{1\leq i}t^iy^i,y)$ which doesn't make sense.

Let $X\to B$ and $Y\to B$ be two flat, projective $B$-schemes ($S$ is already taken ;-). Then

Let $\mathscr Hom_B(X,Y)$ be the functor defined by $$\mathscr Hom_B(X,Y)(Z)=\{Z{\rm -morphisms }\ X\times_B Z\to Y\times_B Z\}.$$ where $Z\to B$ is also a $B$-scheme. Then $\mathscr Hom_B(X,Y)$ is represented by an open $B$-subscheme $${\rm Hom}_B(X,Y)\subset {\rm Hilb}_B(X\times_BY).$$

The $\mathscr Hom$ functor has a subfunctor $\mathscr Isom$ consisting of those morphisms that define a relative isomorphism. This is represented by an open subscheme $${\rm Isom}_B(X,Y)\subset {\rm Hom}_B(X,Y).$$

Now if $B$ is a point, $X=Y$, then this $\rm Isom$ scheme can be identified with the automorphism group of $X$.