How do I unravel apparent recursion in an edef statement?
First some general comments about your code:
In LaTeX use \bfseries instead of \bf and \ttfamily instead of \tt. The 2-letter font changing commands are obsolete since LaTeX2ε was released more than 20 years ago.
\global\edef can be shortened to \xdef. However, a macro does not create a group level, so since you are using this all without any grouping layer, the \global assignments are unnecessary here (unless, of course, in your actual code you need it, then ignore this).
One trick for printing command names is that what you achieve with \ttt command you can do with \ttfamily\string\command or, if you define \def\ttt{\ttfamily\string}, then \ttt\command.
One quite important thing is: do not allocate a register more than once. You are just depleting the number of available registers. Allocate the registers once an use them as many times as you need. I moved \newcount\chNum outside the macro definition.
In number assignments TeX already understands \chNum=`#1 to get the numerical value of an alphabetic constant; you don't need (but it doesn't hurt either) \numexpr here. Also the extra pair of parentheses around `#1 in the \numexpr aren't needed here.
Now on to \testAppendOne. The code
\chNum=\numexpr(`#1)%
\chardef\myCh=\chNum%
\appendChar{\char\myCh}%
appends the two tokens \char\myCh to \resultStr. However, since the \char primitive is not expandable the contents of \resultStr are \char\myCh after the first iteration, \char\myCh\char\myCh after the second one, and so on. When you ask TeX to write the contents of \resultStr it will use \char\myCh\char\myCh... and \myCh will have the last value assigned to it, which will be the last letter you processed, thus the behaviour you see.
When you process a \myCh is \char"61, so \char\char"61 will be a. Pretty obvious. When you process b \myCh is \char"62 then \char\char"62\char\char"62, which is bb and so on. To fix this you have to give \appendChar something that will not change if the \myCh change. The code
\chNum=\numexpr(`#1)%
\chardef\myCh=\chNum%
\appendChar{\char\myCh}%
is redundant. You don't need to \chardef a character before passing it to \char. As you have yourself done for the backslash with \char92, you can just use the number here:
\chNum=`#1
\appendChar{\char\chNum}%
or even:
\appendChar{\char`#1}%
This will pass \char`<some number which will not change> to \appendChar and you'll get what you want.
But why did \char\the\myCh in \testAppendTwo work? Because the primitive \the will access the value in \myCh. Suppose \myCh is \char"61 (a), then \the\myCh will be "61 (hexadecimal) or 97. Thus the code
\chNum=\numexpr(`#1)%
\chardef\myCh=\chNum
\showthe\myCh
\appendChar{\char\the\myCh}%
is also redundant because \the\myCH will access the number in \myCh, which is \chNum, so you can simplify to:
\chNum=\numexpr(`#1)%
\appendChar{\char\chNum}%
\testAppendThree is the same, except that \the\chNum will get the value in the \count register instead of the \char. But they are the same, thus the result is the same (and is the same as the modified version of \testAppendTwo :).
Now to the second part how to debug TeX code... Well... You don't really want to enter here. You'll be happier if you don't (just kidding... or am I?).
You code is rather straightforward, so I managed to debug it using only \show and \showthe. They both are used as \show<macro> and \showthe<register>.
\show prints the contents of a macro to the console. For example, if I change your \appendChar macro to:
\def\appendChar#1{% Append argument character to string
\xdef\resultStr{\resultStr#1}%
\show\resultStr
\par{\ttfamily~\RLp#1\RRp~~~\RLb\resultStr\RRb}\par%
}
and run the \testAppendOne TeX will show this in the terminal:
> \resultStr=macro:
->\char \myCh .
\appendChar #1->\xdef \resultStr {\resultStr #1}\show \resultStr
\par {\ttfamily ~\RLp #1\RRp ~~~\RLb \resultStr \RRb }\par
l.63 \useMacro{\testAppendOne}
?
then, if I press <return>, it shows:
> \resultStr=macro:
->\char \myCh \char \myCh .
\appendChar #1->\xdef \resultStr {\resultStr #1}\show \resultStr
\par {\ttfamily ~\RLp #1\RRp ~~~\RLb \resultStr \RRb }\par
l.63 \useMacro{\testAppendOne}
?
(notice the second line). This showed me why your \testAppendOne was printing the same letter multiple times as I described earlier.
\showthe is similar, but it shows the value of a register. For instance, if you want to see the value of the \chNum register you'd use \showthe\chNum.
These two macros have equivalents which, instead of printing to the console (and the log) print to the PDF. \meaning<macro> will print the definition of the <macro> and \the<register> will print its value.
Not enough debugging?
You can use the \tracingall macro, which tells TeX “give me all you have to show” (and then \tracingnone when you want it to stop), and then your console will be flooded with text. With some practice you'll learn how to read that.
You can also load the trace package and use \traceon/\traceoff, which will hide some steps of the code which will print rather long trace text, like LaTeX's font selection system.
Still not enough?
Load \usepackage{unravel} (after all you asked for unravel :) and use \unravel{<code which is keeping you awake at night>}, and the package will show you step-by-step what TeX is doing. It's pretty useful when you are stuck at some nasty piece of code. Beware that this step-by-step is really thorough, so it might take a handful steps to \unravel{\useMacro{\appendChar}} :)
You are using
\chardef\myCh=\chNum%
\appendChar{\char\myCh}%
neither \char nor a \chardef defined token such as \myCh is expandable so your repeated \edef just produces
\char\myCh\char\myCh\char\myCh\char\myCh
and when you finally use the macro you just get multiple copies of the last defined \myCh
You can debug by adding something like
\texttt{\meaning\resultStr}
to the output. If you do it on your original code you'd get, for \testAppendOne,
For \testAppendTwo and \testAppendThree you'd get
which is not what you want either, is it?
Here's a fixed version of \testAppendOne that appends the real character with a \lowercase trick, namely
\def\testAppendOne#1{% Use a better approach; WORKS
\chNum=\numexpr(`#1)\relax
% Do arithmetic on the counter to implement the sorting rules
\begingroup\lccode`!=\chNum\lowercase{\endgroup\appendChar{!}}%
}
Note other fixes: \ttfamily is used instead of \tt and similarly \bfseries for \bf; more important, you should not use \newcount\chNum at each call: you're wasting a count register at each macro call. I also streamlined the initial macros.
Why do I use \lccode`!=\chNum? The character ! has category code 12, so we essentially normalize all category codes to 12, which seems appropriate when dealing with strings.
\documentclass{article}
\usepackage[svgnames]{xcolor}% to get named colors
\newcommand{\Rtt}[1]{\textcolor{Red}{\ttfamily#1}}
\newcommand\RLp{\Rtt{(}}\newcommand\RRp{\Rtt{)}} % input
\newcommand\RLb{\Rtt{[}}\newcommand\RRb{\Rtt{]}} % [output]
\newcommand\ttt{{\ttfamily\char92}} % A better \tt \textbackslash
\newcount\chNum % NOT inside macros
\def\appendChar#1{% Append argument character to string
\global\edef\resultStr{\resultStr#1}%
\par{\ttfamily~\RLp#1\RRp~~~\RLb\resultStr\RRb}\par
}
\def\testAppendOne#1{% Use a better approach; WORKS
\chNum=\numexpr(`#1)\relax
% Do arithmetic on the counter to implement the sorting rules
\begingroup\lccode`!=\chNum\lowercase{\endgroup\appendChar{!}}%
}
\def\testAppendTwo#1{% Use counter, \chardef and \the; DOESN'T REALLY WORK
\chNum=\numexpr(`#1)\relax
% Do arithmetic on the counter to implement the sorting rules
\chardef\myCh=\chNum
\appendChar{\char\the\myCh}%
}
\def\testAppendThree#1{% Bypass \chardef, DOESN'T REALLY WORK
\chNum=\numexpr(`#1)\relax
% Do arithmetic on the counter to implement the sorting rules
\appendChar{\char\the\chNum}%
}
\def\useMacro#1{% Calls macro #1{<character>}
\global\edef\resultStr{}% (Re)initialize to empty string:
\medskip
\par{\tt\RLp chr\RRp~~\RLb\ttt resultStr\RRb}\par
#1{a}% Simulate character data from string parsing macro
#1{b}%
#1{c}%
#1{d}%
#1{e}%
\medskip
\texttt{\meaning\resultStr}% for debugging
\medskip
}
\begin{document} % B E G I N D O C U M E N T
% Proof of principle of using \edef to rebuild a string
\noindent{\Large\bfseries\color{Blue}Using {\ttt resultStrAppend}:}
\useMacro{\appendChar}
% First attempt fails with apparent recursion
\bigskip\noindent{\Large\bfseries\color{Blue}Using {\ttt testAppendOne}:}
\useMacro{\testAppendOne}
% Second attempt: adding \the command eliminates recursion
\bigskip\noindent{\Large\bfseries\color{Blue}Using {\ttt testAppendTwo}:}
\useMacro{\testAppendTwo}
% Third (final) macro: don't need the \chardef, have working macro
\bigskip\noindent{\Large\bfseries\color{Blue}Using {\ttt testAppendThree}:}
\useMacro{\testAppendThree}
\end{document}
