Can't get unicode symbols in math mode

This is not a problem of cyrillic math characters; if the text were English, the correct input would be

Letters only
\[
\text{Memory: } M_{\textup{add}}(n) = \Theta(N)
\]

because Память and доп are not math. The difference becomes clear when comparing this with the output of

\[
Memory: M_{add}(n) = \Theta(N)
\]

enter image description here

The bottom formula is clearly wrong. Textual subscripts are not math variables, so they should be typeset in the normal text font (upright), thus either \textnormal or \textup (the latter is shorter). Of course, you can define your own command for them.

Here's the complete example:

\documentclass[12pt]{book}

\usepackage{amsmath,amssymb}
\usepackage{unicode-math}
\usepackage{polyglossia}
\setdefaultlanguage[spelling=modern]{russian}
\setotherlanguage{english}

\setmainfont{CMU Serif}
\setsansfont{CMU Sans Serif}
\setmonofont{CMU Typewriter Text}  

\usepackage{color}
\usepackage{minted}
\usepackage[russian]{hyperref}

\setmathfont{Latin Modern Math}

\frenchspacing

\begin{document}

Просто буквы % Plain letters
\[
\text{Память: } M_{\textnormal{доп}}(n) = \Theta(N)
\]
\end{document}

enter image description here

It would be a different problem if you wanted to use a cyrillic letter as a math variable, but your case is not this one.

If you need cyrillic letters as math variables, here's a way to set them up:

\documentclass[12pt]{book}

\usepackage{amsmath,amssymb}
\usepackage{unicode-math}
\usepackage{polyglossia}
\setdefaultlanguage[spelling=modern]{russian}
\setotherlanguage{english}

\setmainfont{CMU Serif}
\setsansfont{CMU Sans Serif}
\setmonofont{CMU Typewriter Text}  

\usepackage[russian]{hyperref}

\setmathfont{Latin Modern Math}

\DeclareSymbolFont{cyrletters}{\encodingdefault}{\familydefault}{m}{it}
\newcommand{\makecyrmathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symcyrletters\endcsname\space #1
}
\count255="409
\loop\ifnum\count255<"44F
  \advance\count255 by 1
  \makecyrmathletter{\count255}
\repeat

\begin{document}
\[
(д+ф)^{2}=д^{2}+2дф+ф^{2}
\]
\end{document}

enter image description here

What does \makecyrmathletter do? Let's review it. The idea is that it takes as argument an integer and performs some magic. We use it in the following loop where the first cycle is

\makecyrmathletter{\count255}

with \count255 having the value "410 (hexadecimal), which corresponds to А U+0410 CYRILLIC CAPITAL LETTER A.

In order to understand the code, I'll assume the explicit value is passed. The first level expansion is then

\begingroup\lccode`a="410\lowercase{\endgroup
\Umathcode`a}="0 \csname symcyrletters\endcsname\space "410

The strange \begingroup construction is used to obtain the letter from the number: we can loop through numbers, not letters. So inside the group, the \lccode of the letter a (the backtick notation is called “alphabetic constant”) to "410. With this setting, \lowercase will scan through its argument, changing every character token into its “lowercase” counterpart, but it actually uses the \lccode table. Then the result will be delivered to be scanned again. Hence we obtain

\endgroup\Umathcode`А="0 \csname symcyrletters\endcsname\space "410

(only the a is changed, control sequences pass through \lowercase with no change). The \endgroup does its job, namely to revert the change of \lccode`a to what it was before, and vanishes.

Then the \Umathcode assignment is performed. It assigns А a math code, that is a new interpretation when found in math mode. The = should be followed by three numbers. The first one states the type of the object; 0 means an ordinary symbol; the second one tells XeTeX from what font family to take it. \csname symcyrletters\endcsname produces the number that has been assigned with the previous \DeclareSymbolFont declaration. Using a symbolic name we don't need to know what number is actually assigned. The third number tells XeTeX what slot the character should be taken from and we obviously choose "410, so a Cyrillic А. The three numbers should be separated by a space, which is explicit in the first case; we need \space in the second case, because leaving a blank space would not work. Since expansion is performed when looking for numbers, this \space is transformed in an actual space token.

A simpler loop can be used with expl3:

\ExplSyntaxOn

\int_step_inline:nnn { "410 } { "44F }
 {
  \Umathcode #1 = "0 ~ \use:c{ symcyrletters } ~ #1
 }

\ExplSyntaxOff

There has been some discussion about Cyrillic in math but the issue is still open: https://github.com/wspr/unicode-math/issues/29

As Joseph mentioned you need fonts with the glyph. In case of math (if you don't switch to a text font with \text{..}) you also need to set mathcodes. E.g.;

\documentclass[12pt]{book}
\usepackage{unicode-math}
\setmainfont{Arial Unicode MS} %for text
\setmathfont[]{xits-math.otf}
\ExplSyntaxOn
\makeatletter
\newcommand\addmathletter[1]{%
  \Umathcode  #1="\mathchar@type\mathalpha \csname sym\um_symfont_tl\endcsname #1\relax
}
\int_step_inline:nnnn {1024}{1}{1154}{
  \addmathletter{#1}
}
\ExplSyntaxOff

\begin{document}
Просто буквы % Plain letters
$$Память: M_{доп}(n) = \Theta(N)$$ % Memory: M_add(n) = \Theta(n)
\end{document}

enter image description here

(The \Umathcode line is copied and adapted from an old message in the xetex list and I don't what the quoting sign at the begin is doing there ...)


Just for fun - in the general case:

general algebra

MWE

\documentclass[12pt]{article}

\usepackage{amsmath,amssymb}
\usepackage{unicode-math}

\setmainfont{Liberation Serif}

\setmathfont{XITS Math}



%See: https://tex.stackexchange.com/questions/201239/cant-get-unicode-symbols-in-math-mode

\DeclareSymbolFont{cyrletters}{\encodingdefault}{\familydefault}{m}{it}
\newcommand{\makecyrmathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symcyrletters\endcsname\space #1
}
\count255="409
\loop\ifnum\count255<"44F
  \advance\count255 by 1
  \makecyrmathletter{\count255}
\repeat

%-----------
\setmainfont{Noto Serif Armenian}
%\familydefault = \rmdefault
\DeclareSymbolFont{armletters}{\encodingdefault}{NotoSerifArmenian(0)}{m}{n}
\newcommand{\makearmmathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symarmletters\endcsname\space #1
}
\count255="530
\loop\ifnum\count255<"587
  \advance\count255 by 1
  \makearmmathletter{\count255}
\repeat



%-----------
\setmainfont{Noto Serif Georgian}
\DeclareSymbolFont{geoletters}{\encodingdefault}{NotoSerifGeorgian(0)}{m}{n}
\newcommand{\makegeomathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symgeoletters\endcsname\space #1
}
\count255="109F
\loop\ifnum\count255<"10FA
  \advance\count255 by 1
  \makegeomathletter{\count255}
\repeat



%-----------
\setmainfont{Noto Serif Lao}
\DeclareSymbolFont{laoletters}{\encodingdefault}{NotoSerifLao(0)}{m}{n}
\newcommand{\makelaomathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symlaoletters\endcsname\space #1
}
\count255="0E80
\loop\ifnum\count255<"0EDF
  \advance\count255 by 1
  \makelaomathletter{\count255}
\repeat



%-----------
\setmainfont{Noto Sans Egyptian Hieroglyphs}
\DeclareSymbolFont{egyletters}{\encodingdefault}{NotoSansEgyptianHieroglyphs(0)}{m}{n}
\newcommand{\makeegymathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symegyletters\endcsname\space #1
}
%\count255="13000
%\loop\ifnum\count255<"1342E %too many?
%  \advance\count255 by 1
%  \makeegymathletter{\count255}
%\repeat
% has 1000 glyphs
\makeegymathletter{"13000}
\makeegymathletter{"13068}
\makeegymathletter{"1307B}
\makeegymathletter{"130D8}
\makeegymathletter{"131C1}



%-----------
\setmainfont{Noto Serif}


\begin{document}

Cyrillic:

\[
(д+ф)^{2}=д^{2}+2дф+ф^{2}м
\]

Armenian:

\[
(է+թ)^{2}=գ^{2}+2ե+ճդ^2-ա
\]

Georgian:

\[
(დ+ლ)^{2}=შ^{2}+2ლ+დშ^2-ა
\]

Lao:

\[
(ມ+ວ)^{2}=ມ^{2}+2ນ+ສວ^2-ກ
\]

Egyptian Hieroglyphs:

\[
(+)^{2}=^{2}+2+^2-
\]

\end{document}

And, of course, symbols can be combined:

for fun two

Here is a way using fontspec package's NFSSFamily= font option to refer to the font families, rather than relying on the internal auto-generated name. And the other font options, such as colour and scale, are also available.

MWE

\documentclass[12pt]{article}
\usepackage{xcolor}
\usepackage{amsmath,amssymb}
\usepackage{unicode-math}

\setmainfont{Liberation Serif}

\setmathfont{XITS Math}

\newfontfamily\fcyr{Noto Serif}[Colour=blue,NFSSFamily=mycyr]
\newfontfamily\farm{Noto Serif Armenian}[Colour=red,NFSSFamily=myarm]
\newfontfamily\fgeo{Noto Serif Georgian ExtraBold}[Colour=brown,NFSSFamily=mygeo]
\newfontfamily\flao{Noto Serif Lao}[Scale=1.2,NFSSFamily=mylao]
\newfontfamily\fegy{Noto Sans Egyptian Hieroglyphs}[Colour=blue,Scale=1.1,NFSSFamily=myegy]

%See: https://tex.stackexchange.com/questions/201239/cant-get-unicode-symbols-in-math-mode

\DeclareSymbolFont{cyrletters}{\encodingdefault}{mycyr}{m}{it}
\newcommand{\makecyrmathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symcyrletters\endcsname\space #1
}
\count255="409
\loop\ifnum\count255<"44F
  \advance\count255 by 1
  \makecyrmathletter{\count255}
\repeat

%-----------
\DeclareSymbolFont{armletters}{\encodingdefault}{myarm}{m}{n}
\newcommand{\makearmmathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symarmletters\endcsname\space #1
}
\count255="530
\loop\ifnum\count255<"587
  \advance\count255 by 1
  \makearmmathletter{\count255}
\repeat



%-----------
\DeclareSymbolFont{geoletters}{\encodingdefault}{mygeo}{m}{n}
\newcommand{\makegeomathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symgeoletters\endcsname\space #1
}
\count255="109F
\loop\ifnum\count255<"10FA
  \advance\count255 by 1
  \makegeomathletter{\count255}
\repeat



%-----------
\DeclareSymbolFont{laoletters}{\encodingdefault}{mylao}{m}{n}
\newcommand{\makelaomathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symlaoletters\endcsname\space #1
}
\count255="0E80
\loop\ifnum\count255<"0EDF
  \advance\count255 by 1
  \makelaomathletter{\count255}
\repeat



%-----------
\DeclareSymbolFont{egyletters}{\encodingdefault}{myegy}{m}{n}
\newcommand{\makeegymathletter}[1]{%
  \begingroup\lccode`a=#1\lowercase{\endgroup
  \Umathcode`a}="0 \csname symegyletters\endcsname\space #1
}
%\count255="13000
%\loop\ifnum\count255<"1342E %too many?
%  \advance\count255 by 1
%  \makeegymathletter{\count255}
%\repeat
% has 1000 glyphs
\makeegymathletter{"13000}
\makeegymathletter{"13068}
\makeegymathletter{"1307B}
\makeegymathletter{"130D8}
\makeegymathletter{"131C1}



%-----------


\begin{document}

Cyrillic:

\[
(д+ф)^{2}=д^{2}+2дф+ф^{2}м
\]

Armenian:

\[
(է+թ)^{2}=գ^{2}+2ե+ճդ^2-ա
\]

Georgian:

\[
(დ+ლ)^{2}=შ^{2}+2ლ+დშ^2-ა
\]

Lao:

\[
(ມ+ວ)^{2}=ມ^{2}+2ນ+ສວ^2-ກ
\]

Egyptian Hieroglyphs:

\[
(+)^{2}=^{2}+2+^2-
\]

Combined:

\[
(_{ມ^უ}+)^{2}=^{2}+\frac{2}{է}+ф(Զ^2)-Ⴔ
\]

\end{document}