How are functors in Haskell related to functors in category theory?

Yes, all Functor instances are endofunctors on Hask--in fact, endofunctors from all of Hask to a proper subcategory whose objects are the types obtained by applying a particular type constructor. That type constructor is what the Functor instance is associated with, and gives the mapping for objects; the mapping for morphisms is fmap, which (because we're only concerned with endofunctors on a cartesian closed category) is itself a family of morphisms in Hask.

It does make sense to consider other functors besides those which can have Functor instances, such as contravariant functors (from Hask to its opposite category). The arr function in the Arrow class also corresponds to a functor, from all of Hask to the category whose objects are the same as those of Hask and whose morphisms are described by the type constructor the Arrow instance is associated with.

Further generalizations are also possible (as Daniel Wagner notes), but tend to become increasingly awkward to use.

An instance of Functor specifies two things: a type constructor F of kind * -> *, that is, a mapping from objects of Hask to objects of Hask, and a function of type (a -> b) -> (F a -> F b), that is, a mapping from arrows of Hask to arrows of Hask compatible with the object-mapping F. So, yes, all instances of Functor are endofunctors. There are several generalizations available on Hackage, e.g. Control.Categorical.Functor.

One important point about this is that what you really want is functors enriched in Hask, not just plain old functors. Hask is cartesian closed (not really, but it tries hard to be onesuch), and so it is naturally enriched in itself.

Now, enriched endofunctors give you a way of restricting to those implementable within the language: an enriched functor Hask -> Hask is a function at the level of objects (types) f a and for each pair of objects a, b a morphism in Hask going f : Hask(a,b) -> Hask(fa,fb). Of course, this is just fmap :: (a -> b) -> f a -> f b