Underbrace under square root sign plain TeX

You could (a) apply a \smash instruction to the \underbrace{...}_{...} part and (b) insert a \vphantom directive after the \sqrt{...} part.

$$
      \sqrt{
         \pmatrix{A^{-1} \cr -1}^{\!T} 
         \smash{\underbrace{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}%
                _{\textstyle A_{\Phi, X\cup\{x\}}}}
         \pmatrix{A^{-1} b(x) \cr -1}
      }
      \vphantom{\underbrace{\pmatrix{.\cr .}}_{\textstyle A_{\Phi}}}
$$
\bye

Observe that I've also dropped some \displaystyle directives and have converted others to \textstyle.

TeX can measure the object:

$$
\def\contentsA{\pmatrix{A^{-1} b(x) \cr -1}}
\def\contentsB{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}
\setbox0=\hbox{$\displaystyle
  \contentsA^{\!T}
  {\underbrace{\!\contentsB\!}_{\displaystyle A_{\Phi, X\cup\{x\}}}}
  \contentsA
$}
\sqrt{\vphantom{\contentsA^T}\hphantom{\copy0}}
\kern-\wd0
\box0
$$

\bye

I made a couple of changes: braces like {\underbrace{...}_{...}} in order to avoid it being considered a math operator with an excess of spacing; for the same reason I introduced a couple of \! tokens. And also one for the exponent in order to place it nearer the top of the parenthesis.