Grothendieck Topos

Giraud's axioms do explicitly refer to the category of sets, through every use of the term "small". E.g. taking the formulation at ncatlab, the category of sets is explicitly referred to by the requirement that $E$ be locally small, and that $E$ have all small coproducts.

I don't know nearly as much about it as I like, but I understand you can replicate broad swaths of the theory of Grothendieck toposes starting from any base topos $S$: you can talk about internal sites, internal presheaves, and internal sheaves, and then the toposes $E$ that comes with a bounded geometric morphism $E \to S$ are precisely those toposes that are equivalent to a categories of internal sheaves over an internal site. e.g. see base topos and bounded geometric morphism from ncatlab.

I don't know what the relative version of Giraud's axioms are, but I imagine they would look quite similar.