How to number subfigures in Serbian Cyrillic?

In [russian]{babel} there is a mapping for cyrillic alphabetical numbering. Since the question asked for polyglossia I copied the definition of that mapping. Then you can redefine the subfigure counter to use the mapping.

MWE:

\documentclass[a4paper,oneside,12pt,serbianc]{article}
% Serbian language and Cyrillic script
\usepackage{fontspec}
\defaultfontfeatures{Mapping=tex-text}
\usepackage{polyglossia}
\setmainlanguage[Script=Cyrillic]{serbian}
\setmainfont{DejaVu Serif}
\newfontfamily\cyrillicfont{DejaVu Serif}[Script=Cyrillic]
\let\cyrillicfonttt\ttfamily
\makeatletter
% mapping copied from Russian babel
% Russian
%\def\cyralph#1{\ifcase #1\or а\or б\or в\or г\or д\or е\or ж\or з\or и\or к\or л\or м\or н\or о\or п\or р\or с\or т\or у\or ф\or х\or ц\or ч\or ш\or щ\or э\or ю\or я\else \@ctrerr \fi}
% Serbian
\def\cyralph#1{\ifcase #1\or а\or б\or в\or г\or д\or ђ\or е\or ж\or з\or и\or ј\or к\or л\or љ\or м\or н\or њ\or о\or п\or р\or с\or т\or ћ\or у\or ф\or х\or ц\or ч\or џ\or ш\else \@ctrerr \fi}
\makeatother
\usepackage{graphicx}
\usepackage[subrefformat=parens,labelformat=parens]{subfig}
% use the mapping for subfigure numbers
\def\thesubfigure{\cyralph{\value{subfigure}}}

\begin{document}
    \begin{figure}
        \centering
        \subfloat[Нешто]{\includegraphics[width=0.4\textwidth]{example-image-a}\label{subfig:abcd}}\quad
        \subfloat[Нешто друго]{\includegraphics[width=0.4\textwidth]{example-image-b}\label{subfig:xyzw}}\\
        \subfloat[Нешто]{\includegraphics[width=0.4\textwidth]{example-image-a}\label{subfig:three}}\quad
        \subfloat[Нешто друго]{\includegraphics[width=0.4\textwidth]{example-image-b}\label{subfig:four}}\\
        \caption{Опис слике}\label{fig:figure}
    \end{figure}

Слика \ref{fig:figure} садржи \subref*{subfig:abcd} и \subref*{subfig:xyzw}.
\end{document}

Note that I changed the font because I don't have Times New Roman here.

Result:

It shouldn't be difficult to add this to gloss-serbian.ldf, similarly to what Russian does.

\documentclass[a4paper,12pt]{article}
% Serbian language and Cyrillic script
\usepackage{fontspec}
\usepackage{polyglossia}

\usepackage{graphicx}
\usepackage[subrefformat=parens,labelformat=parens]{subfig}

\setmainlanguage[Script=Cyrillic]{serbian}
\setmainfont{Times New Roman}
\newfontfamily\cyrillicfont{Times New Roman}[Script=Cyrillic]
\let\cyrillicfonttt\ttfamily

\makeatletter
\def\serbian@Alph#1{\ifcase#1\or
   А\or Б\or В\or Г\or Д\or Ђ\or Е\or Ж\or
   З\or И\or Ј\or К\or Л\or Љ\or М\or Н\or
   Њ\or О\or П\or Р\or С\or Т\or Ћ\or У\or
   Ф\or Х\or Ц\or Ч\or Џ\or Ш\else\xpg@ill@value{#1}{serbian@Alph}\fi}
\def\serbian@alph#1{\ifcase#1\or
   а\or б\or в\or г\or д\or ђ\or е\or ж\or
   з\or и\or ј\or к\or л\or љ\or м\or н\or
   њ\or о\or п\or р\or с\or т\or ћ\or у\or
   ф\or х\or ц\or ч\or џ\or ш\else\xpg@ill@value{#1}{serbian@alph}\fi}

% save the original
\let\latin@alph\@alph
\let\latin@Alph\@Alph
% renew
\let\@alph\serbian@alph
\let\@Alph\serbian@Alph
\makeatother


\begin{document}

The alphabet (should be right):

\newcounter{temp}
\renewcommand{\thetemp}{\Alph{temp}}

\loop\ifnum\value{temp}<30 \stepcounter{temp}\thetemp~\repeat

\setcounter{temp}{0}
\renewcommand{\thetemp}{\alph{temp}}

\loop\ifnum\value{temp}<30 \stepcounter{temp}\thetemp~\repeat


\begin{figure}[htp]
\centering
\subfloat[Нешто]{\includegraphics[width=0.4\textwidth]{example-image-a}\label{subfig:abcd}}\quad
\subfloat[Нешто друго]{\includegraphics[width=0.4\textwidth]{example-image-b}\label{subfig:xyzw}}

\caption{Опис слике}\label{fig:figure}

\end{figure}

Слика \ref{fig:figure} садржи \subref*{subfig:abcd} и \subref*{subfig:xyzw}.

\end{document}