Diagonal brace with nested parenthesis
Very easy with horizontal brace; somewhat less easy with rotated brace. I have no time right now to implement a general solution, so I'll go with a brute-force version for now.
\documentclass{article}
\usepackage{mathtools}% loads amsmath and graphicx
\begin{document}
\[
\biggl(\Bigl(\bigl(a
{% extra group for correct spacing
\overbrace{
{}^q \bigr)^q \Bigr)^{\rotatebox{20}{$\scriptstyle\dots$}} \biggr)^{\!\!q\mathstrut}
}^{\text{$n$ times}}
}
\]
\[
\biggl(
\Bigl(
\bigl(a^q\bigr)^{q}
\Bigr)\quad
\biggr)^{
\!\! q
\setbox0=\hbox to 0pt{\hss
\rotatebox{15}{$
\overbrace{\kern1.8em\scriptstyle\mathstrut\dots\kern1.2em}^{\rotatebox{-15}{\scriptsize$n$ times}}
$}%
}%
\lower1.5ex\box0
}
\]
\end{document}
As given in my answer at Nested Mathematical Functions
The .4pt
stacking gap added with each nesting will determine the growth rate of the parens.
\documentclass{article}
\usepackage{scalerel,stackengine,graphicx,amsmath}
\stackMath
\ignoremathstyle
\newcommand\Comp[1]{\scaleleftright{(}{\addstackgap[.4pt]{#1}}{)}}
\newcommand\rdots{\ThisStyle{.\raisebox{.1ex}{$\scriptstyle.$}%
\raisebox{.2ex}{$\scriptstyle.$}}}
\begin{document}
\[
\stackinset{r}{1pt}{t}{-14.5pt}{\rotatebox{6}
{$\overbrace{\rule{30pt}{0pt}}^{\textrm{\tiny$n$ times}}$}}{%
\Comp{\Comp{\Comp{a^q}^q}^{\rdots}}^q}
\]
\end{document}
Here's a version where the stackgap is increased to .6pt
, to bring out the increasing size of parens:
A solution with pstricks-add
:
\documentclass{article}
\usepackage{pstricks-add}
\begin{document}
\[
\biggl(\Bigl(\bigl(a^{\pnode[0,1ex]{E}{q}}\bigr)^{\! \rnode{F}{\scriptstyle q}}\Bigr)\enspace \biggr)^{\!\!\rnode{G}{\scriptstyle q}\pnode[0,1ex]{H}}%
\psset{singleline, braceWidth=1.2pt, braceWidthInner=2pt, braceWidthOuter=2pt, nodesepB=-2pt}
\psbrace[rot=14,ref=c, linewidth=0.3pt](H)(E){\scriptstyle n\:\mathrm{times}}%
\ncline[linestyle=dotted, dotsep=2.pt, nodesepA=4.5pt, nodesepB=3.2pt]{F}{G}
\]
\end{document}