Equivalence of definitions of ADM Mass

First, let's try and clean up the MTW expression a little bit by getting rid of the manifestly coordinate variant stuff, by introducing a proper contraction, a proper integral over the surface at infinity, and the unit normal $r^{a}$ to the surface at infinity:

$$M_{ADM}={16\pi}\oint\sqrt{\gamma}\,\,d^{2}x \gamma^{ab}r^{c}\left(\gamma_{ac,b}-\gamma_{ab,c}\right)$$

This is only valid, however, not just only for in asymptotically flat spaces, but actually only for asymptotically Cartesian coordinates. You can prove this to yourself by calculating the ``ADM Mass'' of the flat 3-metric in polar coordinates:

$\begin{align*} 16\pi M_{ADM}&=\lim_{r\rightarrow\infty}\oint r^{2}\sin\theta \gamma^{ab}\left(\gamma_{ra,b}-\gamma_{ab,r}\right)d\theta d\phi\\ &=\lim_{r\rightarrow\infty}4\pi r^{2}\left(\gamma^{rr}\gamma_{rr,r}-\gamma^{ab}\gamma_{ab,r}\right)\\ &=\lim_{r\rightarrow\infty}4\pi r^{2}\left(0-\frac{4}{r}\right)\\ &=-\infty \end{align*}$

Obviously, this is wrong, and just an artifact of the way that spherical coordinates behave at infinity. To fix this, we need to always subtract the divergence from the ADM mass of flat spacetime from the ADM mass of the coordinate system in question. So, that's the origin of the $H$ and $H_{0}$ terms. The $H$ term is the extrinsic curvature of the space in question, while $H_{0}$ is the extrinsic curvature of flat spacetime. Now, it's just a matter of showing that the expression inside the integral is equal to $H$.

Typically, I define the extrinsic curvature as $\gamma^{ab}\nabla_{a}r_{b}$. So, let's go on an adventure:

$\begin{align*} H&=\gamma^{ab}\nabla_{a}r_{b}\\ &=\gamma^{ab}\left(\partial_{a}r_{b}-\Gamma_{ab}{}^{c}r_{c}\right)\\ &=\gamma^{ab}\partial_{a}r_{b}-\frac{1}{2}\gamma^{ab}\gamma^{cd}\left(2\gamma_{ad,b}-\gamma_{ab,d}\right)r_{c}\\ &=\gamma^{ab}\partial_{a}\left(\gamma_{bc}r^{c}\right)-\gamma^{ab}r^{c}\left(\gamma_{ac,b}-\frac{1}{2}\gamma_{ab,c}\right)\\ &=\delta^{a}{}_{c}\partial_{a}r^{c} + \gamma^{ab}r^{c}\gamma_{bc,a}-\gamma^{ab}r^{c}\left(\gamma_{ac,b}-\frac{1}{2}\gamma_{ab,c}\right)\\ &=\partial_{a}r^{a}-\frac{1}{2}\gamma^{ab}r^{c}\left(\gamma_{ac,b}-\gamma_{ab,c}\right) \end{align*}$

Now, we note that in any coodinate system adapted so that the coordinate $r=constant$ that determines the surface is chosen for one of the coordinates, we have, necessarily $r^{a}=(1,0,0)$, we note that $\partial_{a}r^{a}=0$, and we thus conclude that the two expressions for ADM mass are equivalent.

Your second expression, when the integral is not taken at infinity, is usually known as either the Brown-York or Hawking-Horowitz (quasilocal) mass. It is closely related to the Liu-Yau mass. For a finite radius sphere, it is not equal to the ADM integral in general.

In the original paper of Brown and York, they asserted but did not provide (as far as I can tell) a detailed proof of the asymptotic behaviour of the quasilocal mass. But if you consider an asymptotically flat space-time and take asymptotic Euclidean coordinates, a direct computation should show that the $K_0\to 0$ as a sphere goes to "infinity" (since coordinate spheres in Euclidean space has mean curvature decaying to 0 as the radius increases), and the decay conditions on the metric and the second fundamental form of the spatial slice should guarantee that the difference between $K$ and the coordinate derivative expression tends to zero. (Remember that the space-time mean curvature can be computed from the mean curvature of the spatial slice embedded in the space-time together with the mean curvature of the two sphere embedded in the spatial slice.)

Similar computations were also performed by Brewin.

The convergence of the Brown-York mass to the ADM mass is also recovered in this paper of Miao, Shi, and Tam.