mathabx + mathtools -> extremely odd underbrace behaviour... how to fix?

use it the other way round:

\usepackage{mathtools}
\usepackage{mathabx}

mathabx has wider effects that aren't always predicted. maybe using just the particular symbols you need, and not loading the entire package, would get rid of this problem (not tried).

good advice can be found at the question Importing a Single Symbol From a Different Font

Mathtools redefines \underbrace and \overbrace, which leads to problems with some fonts. To access the original definitions, use \LaTeXunderbrace and \LaTeXoverbrace, respectively.