Underbrace using STIX2 math font

One should load the mathtools package before the fontspec and/or unicode-math packages. Note: Since the polyglossia package loads fontspec automatically, you should also take care to load mathtools before polyglossia.

For sure, if you move the instruction \usepackage{mathtools} to before both \usepackage{polyglossia} and \usepackage{unicode-math}, the \underbrace stuff works fine, whether or not polyglossia is loaded.

The mathtools package contains a fix to \underbrace and \overbrace as defined in the LaTeX kernel.

If we compare the two definitions

%%% LaTeX kernel
\DeclareRobustCommand\underbrace[1]{\mathop{\vtop{\m@th\ialign{##\crcr
   $\hfil\displaystyle{#1}\hfil$\crcr
   \noalign{\kern3\p@\nointerlineskip}%
   \upbracefill\crcr\noalign{\kern3\p@}}}}\limits}

%%% mathtools.sty
\def\underbrace#1{\mathop{\vtop{\m@th\ialign{##\crcr
   $\hfil\displaystyle{#1}\hfil$\crcr
   \noalign{\kern.7\fontdimen5\textfont2\nointerlineskip}%
   \upbracefill\crcr\noalign{\kern.5\fontdimen5\textfont2}}}}\limits}

we see that they're almost identical, but mathtools improves the definition by not using a fixed length of 3pt and adjusting it with the current math font setup.

However, the definition of \underbrace in unicode-math is

\UnicodeMathSymbol{"023DF}{\underbrace}{\mathunder}{bottom curly bracket (mathematical use)}

that translates to more low level

\underbrace=\protected macro:
#1->\mathop {\Umathaccent bottom 7\symoperators "023DF\scan_stop: {{}#1}}\limits

What happens if you load mathtools after unicode-math is that the “Unicode” definition of \underbrace is overridden with the fix above done by mathtools. Note that you get no error: indeed \upbracefill uses math characters that in unicode-math don't correspond to the pieces of brace as in legacy LaTeX math fonts.

In conclusion: mathtools makes a good fix for legacy math fonts, but it shouldn't do it when unicode-math is loaded. Until mathtools is updated, the solution is to load it before unicode-math.