Is there a test for the different styles inside maths mode?

Lets call such a macro mathsettoheight. I'll keep things simple and define only mathsettoheight (and not mathsettowidth and mathsettodepth).

IIUC, you want this macro to behave like

\mathsettoheight{\dimen}{content}

First, lets consider a simplified version of \settoheight

\def\settoheight#1#2%
  {\setbox\@tempboxa\hbox{{#2}}%
   #1=\ht\@temboxa
   \setbox\@temboxa\box\voidb@x}

The only change that we need to do in \mathsettoheight is to set the box correctly. Instead of

\hbox{#2}

use

\hbox{$\mathpalette{}{#2}$}

\mathpalette is a shortcut used for convenience and is usually defined by all macro packages (plain TeX, LaTeX, ConTeXt) as

\def\mathpalette#1#2{%
  \mathchoice
    {#1\displaystyle{#2}}%
    {#1\textstyle{#2}}%
    {#1\scriptstyle{#2}}%
    {#1\scriptscriptstyle{#2}}}

A minor improvement is to use

\hbox{$\m@th\mathpalette{}{#2}$}

where \m@th sets the math surround to zero (useful if you are interested in the width of the content).

And thats it. The complete definition with you example is

\documentclass{amsart}

\makeatletter

\def\mathsettoheight#1#2%
  {\setbox\@tempboxa\hbox{{#2}}%
   #1=\ht\@temboxa
   \setbox\@temboxa\box\voidb@x}

\def\mathsettoheight#1#2%
  {\setbox\@tempboxa\hbox{$\m@th\mathpalette{}{#2}$}%
   #1=\ht\@tempboxa
   \setbox\@tempboxa\box\voidb@x}

\makeatother

% Test case

\newlength\mathheight
\newcommand*\intheight{%
  \mathsettoheight{\mathheight}{\int}
  \the\mathheight
}


\begin{document}

$\intheight$

\begin{align}
\text{displaystyle}&={\displaystyle\text{\intheight}}\\
 \text{textstyle}&={\textstyle\text{\intheight}} \\
 \text{scriptstyle}&={\scriptstyle\text{\intheight}} \\
 \text{scriptscriptstyle}&={\scriptscriptstyle\text{\intheight}} 
\end{align}

{\obeylines
 \text{displaystyle}= $\displaystyle\text{\intheight}$\\
 \text{textstyle}=$\textstyle\text{\intheight}$ \\
 \text{scriptstyle}=$\scriptstyle\text{\intheight}$ \\
 \text{scriptscriptstyle}=$\scriptscriptstyle\text{\intheight}$ 
\endgraf}

\end{document}

This answer will only apply to LuaTeX and LuaLaTeX, but your question was quite general, so ...

The luatex engine defines a primitive that allows you to query the current math style.

This primitive is called \mathstyle, and when the value is queried (after \ifnum, for example) it returns a number between -1 (not in math mode), 0 (display style), and 7 (cramped scriptscript style).

Luatex also defines four new primitives with \cramped.. prefix: \crampeddisplaystyle, \crampedtextstyle, \crampedscriptstyle, \crampedscriptscriptstyle. These are like their non-prefixed cousins (\displaystyle ... \scriptscriptstyle) except that they explicitly switch to one of the 'cramped' math styles.

Finally, there is the new primitive \Ustack, that is to be used as a prefix for \over.. and \atop.. constructs (which is what commands like \frac and \binom eventually expand into), like so

\Ustack {a \over b}

If you do not do this, \mathstyle will return the wrong value in the initial part of {... \over ...}, as explained by Aditya.

All combined together, these primitives allow code like this:

\ifnum\mathstyle=\textstyle
   \message{normal text style}
\else \ifnum\mathstyle=\crampedtextstyle
   \message{cramped text style}
\fi \fi

and this:

\def\cramped#1% switch the argument to a cramped math style
  {{\ifcase\mathstyle
    \crampeddisplaystyle      \or \or % 0 -> 1
    \crampedtextstyle         \or \or % 2 -> 3
    \crampedscriptstyle       \or \or % 4 -> 5
    \crampedscriptscriptstyle \fi     % 6 -> 7
    #1}}

and even allow you to define a version of \mathchoice that does not typeset everything four times:

\def\mathchoice#1#2#3#4%
  {{\ifcase\mathstyle
    #1\or #1\or
    #2\or #2\or
    #3\or #2\or
    #4\or #4\else #2\fi}}

All this can currently only be used if your underlying engine is luatex, but it is hoped that this extension will eventually become supported by xetex as well.

Yes, you have \mathchoice

\documentclass{article}

\newcommand\mathtest{%
  \mathchoice
  {displaystyle}
  {text style}
  {scriptstyle}
  {scriptscriptstyle}
}

\begin{document}
\[
 \mathtest x^{\mathtest^{\mathtest}}
\]

\(
  \mathtest
\)

\end{document}