\tiny etc. in mathmode

Size commands do not work in math mode. However (some) size commands set up math so if you enter a new math expression from text while the size change is in force, you get math of a matching size. Internally array is an \halign in which each cell is surrounded by $...$ so in fact using array is like using \mbox{empty text $ matrix cell$} so the entries pick up the size change more or less by accident.

\documentclass{scrbook}
\usepackage{amsmath}
\begin{document}
$M_{\begin{smallmatrix}2\\2\end{smallmatrix}2A}$

\end{document}

A simple way to change the font size in math mode is to put the size-changing command inside a \mbox, for example:

\documentclass[varwidth, preview]{standalone}
\usepackage[T1]{fontenc}    % Or unicode-math
\usepackage[utf8]{inputenc} % The default since 2018.

\newcommand\mscriptsize[1]{\mbox{\scriptsize\ensuremath{#1}}}
\newcommand\mtiny[1]{\mbox{\tiny\ensuremath{#1}}}

\begin{document}
\( x \mscriptsize{y} \mtiny{\omega} \)
\end{document}

You might want only the braces if you need a long expression to wrap.

You can also use \text from amsmath to insert arbitrary text-mode commands within math mode, such as \text{\tiny\itshape x}.

You could define a tinymatrix environment modeled on smallmatrix:

\documentclass{article}
\usepackage{amsmath}

\makeatletter
% from amsmath.sty, line 960:
\newenvironment{tinymatrix}{\null\,\vcenter\bgroup
  \Let@\restore@math@cr\default@tag
  \baselineskip4\ex@ \lineskip1.2\ex@ \lineskiplimit\lineskip
  \ialign\bgroup\hfil$\m@th\scriptscriptstyle##$\hfil&&\thickspace\hfil
  $\m@th\scriptscriptstyle##$\hfil\crcr
}{%
  \crcr\egroup\egroup\,%
}
\makeatother

\newcommand{\di}[1]{\begin{tinymatrix}#1\end{tinymatrix}}

\begin{document}

\[
M_{\di{2\\2}2A}
\]

\end{document}