Smaller font for subscript within subscript in math mode

You can use \DeclareMathSizes; declare also other sizes if needed.

\RequirePackage{fix-cm} % arbitrary font scaling
\documentclass{article}

\DeclareMathSizes{10}{10}{7}{4}

\begin{document}
$a_{b_{c}}$
\end{document}

enter image description here

Compare to the standard and decide for yourself if it's worth the trouble. I wouldn't go below four points.

enter image description here


For even smaller, you could scale the sub-subscript:

\documentclass{article}
\usepackage{scalerel}
\begin{document}
$a_{b_{\scaleto{c}{1pt}}}$
\end{document}

enter image description here

If you need consistency across a range of subscripts, you may wish to add a normalizer like a \mathstrut:

\documentclass{article}
\usepackage{scalerel}
\begin{document}
$a_{b_{\scaleto{c\mathstrut}{2pt}}}$
$a_{b_{\scaleto{g\mathstrut}{2pt}}}$
\end{document}

enter image description here

Here, \scaleto from the scalerel package operates on its arguments in math mode by default, unlike \scalebox of the graphicx package, which operates in text mode.


Using someone's idea in How to get an even smaller font? to scale \scriptscriptstyle content down by 30%:

enter image description here

\documentclass{article}

\usepackage{graphicx}

\begin{document}

$a_{b_c}$

$a_{b_{\scalebox{.7}{$\scriptscriptstyle c$}}}$

\end{document}