Underbrace under square root sign plain TeX

You could (a) apply a \smash instruction to the \underbrace{...}_{...} part and (b) insert a \vphantom directive after the \sqrt{...} part.

enter image description here

         \pmatrix{A^{-1} \cr -1}^{\!T} 
         \smash{\underbrace{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}%
                _{\textstyle A_{\Phi, X\cup\{x\}}}}
         \pmatrix{A^{-1} b(x) \cr -1}
      \vphantom{\underbrace{\pmatrix{.\cr .}}_{\textstyle A_{\Phi}}}

Observe that I've also dropped some \displaystyle directives and have converted others to \textstyle.

TeX can measure the object:

\def\contentsA{\pmatrix{A^{-1} b(x) \cr -1}}
\def\contentsB{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}
  {\underbrace{\!\contentsB\!}_{\displaystyle A_{\Phi, X\cup\{x\}}}}


enter image description here

I made a couple of changes: braces like {\underbrace{...}_{...}} in order to avoid it being considered a math operator with an excess of spacing; for the same reason I introduced a couple of \! tokens. And also one for the exponent in order to place it nearer the top of the parenthesis.