\sin in section heading + xelatex + unicode-math fails to produce \tableofcontents

Commands such as \cos (operator names) are fragile, so they should be prefixed by \protect.

With amsmath the problem is milder, because \cos expands to \qopname\relax{o}{cos} and `\qopname is robust.

In your setting, you should do

\AtBeginDocument{%
  \@ifpackageloaded{amsopn}
    {\DeclareMathOperator{\arctg}{arctg}}%
    {\def\arctg{\mathop{\operator@font arctg}\nolimits}}%
}

so as to uniformize the definitions.

You can also make the commands robust:

\documentclass{article}

\usepackage{ifxetex,ifluatex}
\ifxetex
    \usepackage{fontspec}
    \usepackage{unicode-math}
\else\ifluatex
    \usepackage{fontspec}
    \usepackage{unicode-math}
\else
    \usepackage[T1,T2A]{fontenc}
    \usepackage[utf8]{inputenc}
\fi\fi

\makeatletter
\AtBeginDocument{%
  \@ifpackageloaded{amsopn}
    {\DeclareMathOperator{\arctg}{arctg}}%
    {\def\arctg{\mathop{\operator@font arctg}\nolimits}}%
}
\def\protectoperators#1{%
  \AtBeginDocument{%
    \@tfor\next:=#1\do{\expandafter\MakeRobust\next}%
  }
}
\makeatother

\protectoperators{%
  \sin\cos\log\ln\arctg % add the other operators
}


\begin{document}

\show\sin

\ifxetex
    This is Xe\LaTeX
\else\ifluatex
    This is Lua\LaTeX
\else
    This is pdf\LaTeX
\fi\fi

\subsubsection{Function $\arctg(x)$}

Function
\begin{equation}
\arctg(x)
\end{equation}

\subsubsection{Function $\cos(x)$}

\begin{equation}
\cos(x)
\end{equation}

\tableofcontents

\end{document}

Here is the contents of the .toc file:

\contentsline {subsubsection}{\numberline {0.0.1}Function $\arctg (x)$}{1}
\contentsline {subsubsection}{\numberline {0.0.2}Function $\cos (x)$}{1}