What does \smash do, and where is it documented?

As the name implies, \smash takes its contents and prints it as if its height and depth were zero.

The definition in LaTeX is carried over from the one in plain TeX, with a slight difference; I'll use the LaTeX one for ease of reference. The following refers to LaTeX 2018-12-01.

% latex.ltx, line 4494:
\def\smash{%
  \relax % \relax, in case this comes first in \halign
  \ifmmode
    \expandafter\mathpalette\expandafter\mathsm@sh
  \else
    \expandafter\makesm@sh
  \fi}

The macro distinguish if it is called in math mode or not. Note that, for efficiency, it doesn't read its argument at the outset. I'll assume the call is \smash{abq}. In math mode we get

\mathpalette\mathsm@sh{abq}

and we now need to look at \mathsm@sh:

% latex.ltx, line 4503:
\def\mathsm@sh#1#2{%
  \setbox\z@\hbox{$\m@th#1{#2}$}\finsm@sh}

According to the working of \mathpalette, we get the equivalent of

\setbox\z@\hbox{$\m@th<current style>{abq}$}\fin@smash

If the call is in text mode, we have \makesm@sh{abq}. Now

% latex.ltx, line 4501:
\def\makesm@sh#1{%
  \setbox\z@\hbox{\color@begingroup#1\color@endgroup}\finsm@sh}

so we obtain

\setbox\z@\hbox{\color@begingroup abq\color@endgroup}\finsm@sh

In both cases TeX has set the contents of box 0; now \finsm@sh does its job

% latex.ltx, line 4505:
\def\finsm@sh{\ht\z@\z@ \dp\z@\z@ \leavevmode@ifvmode\box\z@}

This sets the height and depth of box 0 to 0pt and typesets the box after initiating horizontal mode.

When you load amsmath the definition is basically the same, with the difference that an optional argument is allowed, which can be either t or b. With t the last operation becomes only

\ht\z@\z@ \box\z@

so the depth is preserved; conversely, with b LaTeX only does

\dp\z@\z@ \box\z@

and the height is preserved. This is accomplished by the redefinition (referring to amsmath.sty release 2018-12-01)

% amsmath.sty, line 903:
\ifx\leavevmode@ifvmode\@undefined
\renewcommand{\smash}[1][tb]{%
  \def\mb@t{\ht}\def\mb@b{\dp}\def\mb@tb{\ht\z@\z@\dp}%
  \edef\finsm@sh{\csname mb@#1\endcsname\z@\z@\box\z@}%
  \ifmmode \@xp\mathpalette\@xp\mathsm@sh
  \else \@xp\makesm@sh
  \fi
}
\else
\renewcommand{\smash}[1][tb]{%
  \def\mb@t{\ht}\def\mb@b{\dp}\def\mb@tb{\ht\z@\z@\dp}%
  \edef\finsm@sh{\csname mb@#1\endcsname\z@\z@ \leavevmode@ifvmode\box\z@}%
  \ifmmode \@xp\mathpalette\@xp\mathsm@sh
  \else \@xp\makesm@sh
  \fi
}
\fi

You see that \finsm@sh is redefined each time to consist of the necessary bits.

The conditional definition of \smash in amsmath.sty is due to the fact that \leavevmode@ifvmode has been added to LaTeX starting from version 2018-12-01 to initiate horizontal mode if found in vertical mode. In prior version of LaTeX this didn't happen, reflecting the same behavior as in plain TeX, where the definition is the same as in the LaTeX kernel, but without \leavevmode@ifvmode, which is defined in the LaTeX kernel by

% latex.ltx, line 1633:
\protected\def\leavevmode@ifvmode{\ifvmode\expandafter\indent\fi}

\smash is a macro defined as

\relax \ifmmode \def \next {\mathpalette \mathsm@sh }\else \let \next \makesm@sh \fi \next 

and \mathsm@sh is defined as:

\setbox \z@ \hbox {$\m@th #1{#2}$}\finsm@sh 

and finsm@sh is defined as

ht \z@ \z@ \dp \z@ \z@ \box \z@ 

A simpler version of the command which makes what it does a bit more obvious is this one from TeX by Topic:

\def\smash#1{{\setbox0=\hbox{#1}\dp0=0pt \ht0=0pt \box0\relax}}

which basically takes some content and puts it into a box of 0 height and 0 depth.

It's documented in the TeXbook and also in TeX by Topic, which is included as part of the TeX Live documentation (texdoc texbytopic should bring it up.)

The actual macros in the LaTeX kernel can be found in the document source2e.pdf (also available through texdoc.)

Some packages such as amsmath redefine it. See also

• Which package defines \smash?