Value of LuaTeX’s \mathstyle in \frac (de)nominator
\frac{a}{b}
is essentially {a \over b}
and the big problem with the infix \over
is that it makes tracking the current style hard luatex does not change that.
to get a version in which \mathstyle
is set you need \Ustack{a \over b}
so in latex you would have to redefine \frac
and related commands to use \Ustack
.
One possibility would be to load lualatex-math
which redefines \frac
in this way.