Precompile header with xelatex

I had the same problem myself once. The first solution is of course, to take \usepackage{fontspec} out of the preamble document and put it right before \begin{document} in the main document.

But as you, just like me, want to load fontspec in a custom document class, you could use

\RequirePackage{etoolbox}
\AtEndPreamble{
    \usepackage{fontspec}
    \setmainfont[Ligatures=TeX]{STIXGeneral}
}

there. \AtEndPreamble works similar to the standard LaTeX command \AtBeginDocument, only that the code inside can itself use \AtBeginDocument like fontspec does.

Unfortunately this means that you can't cut the compile time of fontspec here. The same problem exists with math-unicode and polyglossia for example. (While I had no problems with babel.)

And here is an explanation as to why this is not possible, that I found on the XeTeX mailing list:

Jonathan Kew a écrit :

Right; this would present various technical challenges, and could be quite confusing for the user if the .fmt file contained references to fonts that meanwhile have been removed or modified in the host
system.

Ok. OTOH, I don't understand exactly how font preloading works for "classical" TeX, but the same could happen if the fonts were removed from the texmf tree (or even form the map files)? Or maybe TeX needs only the tfms while XeTeX is using "more" of the font?

Yes, that's basically the situation. With "classical" TeX, the .fmt
file contains the entire tfm data for the preloaded fonts, so TeX can then do its work without reference to the tfms or any other files. (If you remove the tfms, it won't even notice as it already has that data. If you remove the pfbs or map file entries, TeX won't care, it can
still typeset. Your output driver may have problems, though!)

But for the equivalent to work with xetex, we'd have to "embed" a
large portion of the entire OpenType font into the .fmt file. This
doesn't seem like a good idea.

Your preamble loads a ‘native’ (i.e. non TFM) font, which as the meessage says can't be dumped in the format. For example, just having \usepackage{fontspec} in your preamble will load Latin Modern OpenType fonts.