Expand multiple math-tokens before detokenize
In the general case (without knowing the exact composition of the full argument to \detokenize
), you have to expand your way into the argument.
To dissect this example below, the 1st, 3rd, and 4th \expandafter
s expand the 2nd macro once, leaving the 2nd \expandafter
to finally expand the 1st macro once.
Note that no \expandafter
is needed before the \detokenize
.
\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\def\userMacro{\sqrt{4}}
\begin{document}
\texttt{My Verbatim equation:
\detokenize\expandafter\expandafter\expandafter{\expandafter\userMacro\userMacro}}
\end{document}
Now...
if you can guarantee that the argument will only contain macros that you want detokenized exactly once, then this will automate it:
\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\let\edetok\relax
\def\userMacroA{\sqrt{4}}
\def\userMacroB{\frac{A}{B}}
\def\userMacroC{\qed}
\newcommand\xdetokenize[1]{\detokaux#1\edetok\edetok}
\def\detokaux#1#2\edetok{\ifx\edetok#1\edetok\else
\detokenize\expandafter{#1}\def\next{\detokaux#2\edetok\edetok}\expandafter\next\fi
}
\begin{document}
\ttfamily My Verbatim equation:
1: \xdetokenize{}
2: \xdetokenize{\userMacroA}
3: \xdetokenize{\userMacroA\userMacroB}
4: \xdetokenize{\userMacroA\userMacroB\userMacroC}
\end{document}
If you put plain text into the argument of \xdetokenize
, the spaces will be eaten up (I warned you). It could be adapted to avoid that, but becomes increasingly complex to do so.
So you have a snippet of text (or whatever set of tokens) which is interspersed by some tokens that need to be expanded to whatsoever level.
A homespun and old fashioned approach to the matter could be splitting that snippet of text up into pieces whereof each piece begins with a token that is to be expanded.
Then have these pieces nested in reverse order.
Then with each piece have expanded the first token as many times as needed and then have that piece exchanged with the pieces that are nested behind it. This means reversing the reversed order after expanding. This means obtaining the initial order while things are expanded.
Before elaborating on that and delivering a minimal working example, let's in advance have some things related to (La)TeX explained.
First thing:
In (La)TeX there are so-called primitives. Some of them process so called ⟨general text⟩. E.g., \detokenize
doses so.
The Syntax of \detokenize
is:
\detokenize<general text>
while
<general text>=<filler><catcode-1-token><balanced text><explicit character token of catcode 2>
. Thus the syntax of \detokenize
can also be described as:
\detokenize<filler><catcode-1-token><balanced text><explicit character token of catcode 2>
⟨filler⟩ is an arbitrary set of space tokens and tokens whose meaning equals the meaning of the \relax
-primitive. ⟨filler⟩ will be removed.
⟨balanced text⟩ is an arbitrary set of tokens containing explicit catcode-1(begin group) character tokens and explicit catcode-2(end group) character tokens in amounts that are equal to each other.
\detokenize
will extract ⟨balanced text⟩ from ⟨general text⟩ and act on that ⟨balanced text⟩ as if it were written to external file unexpanded and read back and tokenized from that external file under a catcode-régime where the space token has category code 10(space) and all other characters have category code 12(other).
The funny thing is:
When (La)TeX searches for the ⟨catcode-1-token⟩ preceding the ⟨balanced text⟩, it will both keep removing tokens forming the ⟨filler⟩ (i.e., space tokens and tokens whose meaning equals the meaning of the \relax
primitive) and keep expanding exandable tokens until finding the desired ⟨catcode-1-token⟩.
The fact that (La)TeX keeps expanding while removing ⟨filler⟩ and searching for that desired ⟨catcode-1-token⟩ can be used for (e.g., via \expandafter
and \romannumeral
-expansion) triggering a lot of expansion work before letting (La)TeX find that desired ⟨catcode-1-token⟩.
Second thing:
You can use the expandable \romannumeral
-primitive for triggering a lot of expansion work: \romannumeral
usually causes (La)TeX to gather a ⟨number⟩ and deliver its representation in terms of lowercase roman numerals. While gathering tokens considered (digits of) ⟨numbers⟩, (La)TeX keeps expanding expandable tokens. In case the ⟨number⟩ gathered is not positive, the tokens forming that ⟨number⟩ will be "eaten" by (La)TeX while in that case (La)TeX will not deliver any token at all. Therefore one can (ab?)use the \romannumeral
-primitive for triggering a lot of expansion-work (e.g., argument-exchanging-work, token-replacing-work) as long as it is ensured that in the end (La)TeX will find a non-positive ⟨number⟩.
Third thing:
Usually you use \expandafter
for influencing the order in time in which expandable tokens get expanded.
E.g., if you wish the second token of the arrangement of tokens which is formed by the input-stream in chronological order to be expanded before the first token of the input-stream, you just make sure that within the input-stream the first token will be preceded by \expandafter
:
\def\first{1}
\def\second{2}
\expandafter\first\second
-> \first 2
\expandafter
itself is expandable. Referring to the analogy in the TeXbook, where TeX' processing of input and tokens is compared to a digestive tract where expansion takes place in TeX' mouth, one could say that \expandafter
is processed in TeX' mouth. When work triggered by \expandafter
is done, the \expandafter
-token in question will not be there any more. (La)TeX considers the work triggered by \expandafter
done when the process of attempting to expand the next but one token is finished. Therefore you can intersperse the first k elements of a set of tokens with \expandafter
in order to have (La)TeX expand the (k+1)-th token before further processing the first token of that set. Such interspersing is called creating an \expandafter
-chain:
\def\first{1}
\def\second{2}
\def\third{3}
\def\fourth{4}
\expandafter\first\expandafter\second\expandafter\third\fourth
-> \first\second\third 4.
You can have several \expandafter
-chains.
For each additional level of expansion with a specific token as starting-point, you need to add another \expandafter
-chain "leading" towards that token. The amount of \expandafter
-tokens needed increases exponentially with the amount of expansion-levels desired:
\def\first{1}
\def\second{2}
\def\third{3}
\def\fourth{4}
\def\leveluponfourth{\fourth}
%|1st \expand|2nd \expand|%
%|after-chain|after-chain|%
%| | | | |%
\expandafter\expandafter
\expandafter \first
\expandafter\expandafter
\expandafter \second
\expandafter\expandafter
\expandafter \third
\leveluponfourth
->
\expandafter\first
\expandafter\second
\expandafter\third
\fourth
->
\first\second\third 4
You can avoid inserting many \expandafter
-chains by using \romannumeral
-expansion for implementing a mechanism whose syntax is
\romannumeral\Expandtimes{<number whose value is k>}<token-sequence>
and which delivers k "hits" by \expandafter
to the first token of ⟨token-sequence⟩ as a result of triggering expanding the \romannumeral
-primitive. Expanding the \romannumeral
-primitive in turn can, e.g., be triggered by a single \expandafter
-chain "leading" towards that \romannumeral
-primitive.
In expansion contexts only one \expandafter
-chain leading to the \romannumeral
-primitive is needed for obtaining the same result that is obtained by k \expandafter
-chains leading to the first token of the ⟨token-sequence⟩:
\def\first{1}
\def\second{2}
\def\third{3}
\def\fourth{4}
\def\leveluponfourth{\fourth}
\expandafter\first
\expandafter\second
\expandafter\third
\romannumeral\Expandtimes{2}\leveluponfourth
->
\first\second\third 4
There are several ways of implementing \Expandtimes
. One of them is shown in the minimal working example at the end of this answer.
After having explained these things (or at least after having tried to), let's go back to my homespun and old fashioned approach to the matter:
Assume you have the sequence
Whatsoever before tokenA.\tokenA Whatsoever between
tokenA and tokenB.\tokenB Whatsoever between tokenB
and tokenC.\tokenC Whatsoever between tokenC and
tokenD.\tokenD Whatsoever after tokenD.
, and you wish
\tokenA
to be expanded 3 times ,
\tokenB
to be expanded 1 times ,
\tokenC
to be expanded 2 times ,
\tokenD
to be expanded 4 times.
First split that snippet of text up into pieces whereof each piece begins with a token that is to be expanded:
{Whatsoever before tokenA.}%
{\tokenA Whatsoever between tokenA and tokenB.}%
{\tokenB Whatsoever between tokenB and tokenC.}%
{\tokenC Whatsoever between tokenC and tokenD.}%
{\tokenD Whatsoever after tokenD.}%
Then have these pieces nested in reverse order:
{\tokenD Whatsoever after tokenD.}{%
{\tokenC Whatsoever between tokenC and tokenD.}{%
{\tokenB Whatsoever between tokenB and tokenC.}{%
{\tokenA Whatsoever between tokenA and tokenB.}{%
Whatsoever before tokenA.%
}%
}%
}%
}%
Then with each piece have expanded the first token as many times as needed and also have the piece exchanged with the pieces that are nested behind it. In the minimal working example below, all this is done by means of the macro \Expandtimesexchange
whose first argument denotes the level of expansion needed with the first token of its second argument before exchanging its second and its third argument:
\Expandtimesexchange{4}{\tokenD Whatsoever after tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between tokenC and tokenD.}{%
\Expandtimesexchange{1}{\tokenB Whatsoever between tokenB and tokenC.}{%
\Expandtimesexchange{3}{\tokenA Whatsoever between tokenA and tokenB.}{%
Whatsoever before tokenA.%
}%
}%
}%
}%
In case you wish all this to be done before applying \detokenize
, you can –
via inserting \expandafter
right behind \detokenize
– trigger some \romannumeral
-expansion which will be terminated when arguments are expanded and exchanged around in a way where as a result of expansion \romannumeral
will find the non-positive number 0 - the space token behind the zero "lets (La)TeX know" that the digit-sequence to be gathered is complete. Thus that space token will be "eaten" as part of completing the "gathering-the-digits-of-a-number"-process triggered by \romannumeral
:
\detokenize\expandafter{%
\romannumeral\Expandtimesexchange{4}{\tokenD Whatsoever after tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between tokenC and tokenD.}{%
\Expandtimesexchange{1}{\tokenB Whatsoever between tokenB and tokenC.}{%
\Expandtimesexchange{3}{\tokenA Whatsoever between tokenA and tokenB.}{%
0 Whatsoever before tokenA.% "0<space>" ends \romannmeral expansion.
}%
}%
}%
}%
}%
Here is the promised complete minimal working example:
\documentclass[a4paper,11pt]{article}
\begingroup
\makeatletter
\@firstofone{%
\endgroup
\newcommand*\dfork[1]{\innerdfork#1{\@firstoftwo}d{\@secondoftwo}dd}%
}%
\newcommand\innerdfork{}\def\innerdfork#1d#2#3dd{#2}%
\newcommand*\Expandtimes[1]{%
0\expandafter\innerExp
\expandafter{%
\expandafter}%
\romannumeral\number\number#1 000d%
}%
\newcommand*\innerExp[2]{\dfork{#2}{#1 }{\innerExp{#1#1\expandafter}}}%
\newcommand*\Expandtimesexchange[2]{%
\expandafter\bracestripexchange\expandafter{\romannumeral\Expandtimes{#1}#2}%
}%
\newcommand\bracestripexchange[2]{#2#1}%
\newcommand\tokenA{\onelevelA}
\newcommand\tokenB{ [B.] }
\newcommand\tokenC{\onelevelC}
\newcommand\tokenD{\onelevelD}
\newcommand\onelevelA{\twolevelA}
\newcommand\onelevelC{ [C.] }
\newcommand\onelevelD{\twolevelD}
\newcommand\twolevelA{ [A.] }
\newcommand\twolevelD{\treelevelD}
\newcommand\treelevelD{ [D.] }
\begin{document}
\scriptsize
\parindent=0ex
\parskip=\baselineskip
\begin{verbatim}
\newcommand\tokenA{\onelevelA}
\newcommand\tokenB{ [B.] }
\newcommand\tokenC{\onelevelC}
\newcommand\tokenD{\onelevelD}
\newcommand\onelevelA{\twolevelA}
\newcommand\onelevelC{ [C.] }
\newcommand\onelevelD{\twolevelD}
\newcommand\twolevelA{ [A.] }
\newcommand\twolevelD{\treelevelD}
\newcommand\treelevelD{ [D.] }
\expandafter\newcommand
\expandafter\test
\expandafter{%<-these \expandafter trigger \detokenize
\detokenize\expandafter{%<- this \expandafter triggered by \detokenize triggers \romannumeral-expansion
\romannumeral\Expandtimesexchange{4}{\tokenD Whatsoever after \tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between \tokenC and \tokenD.}{%
\Expandtimesexchange{1}{\tokenB Whatsoever between \tokenB and \tokenC.}{%
\Expandtimesexchange{3}{\tokenA Whatsoever between \tokenA and \tokenB.}{%
0 Whatsoever before \tokenA.% "0<space>" ends \romannmeral expansion.
}%
}%
}%
}%
}%
}%
\end{verbatim}
\expandafter\newcommand
\expandafter\test
\expandafter{%<-these \expandafter trigger \detokenize
\detokenize\expandafter{%<- this \expandafter triggered by \detokenize triggers \romannumeral-expansion
\romannumeral\Expandtimesexchange{4}{\tokenD Whatsoever after \tokenD.}{%
\Expandtimesexchange{2}{\tokenC Whatsoever between \tokenC and \tokenD.}{%
\Expandtimesexchange{1}{\tokenB Whatsoever between \tokenB and \tokenC.}{%
\Expandtimesexchange{3}{\tokenA Whatsoever between \tokenA and \tokenB.}{%
0 Whatsoever before \tokenA.% "0<space>" ends \romannmeral expansion.
}%
}%
}%
}%
}%
}%
\texttt{\string\meaning\string\test \(\to\) \meaning\test}
\bigskip
Now see that \texttt{\string\detokenize}-ing took place before definig:
\bigskip
\texttt{\string\test \(\to\) \test}
\end{document}