Question about the special rule mentioned after TeXbook exercise 20.5
With \def\a#1#{\hbox to #1}, a call such as
\a3pt{x}
makes TeX find 3pt as the (delimited) argument. This triggers the replacement and you get
\hbox to 3pt{x}
The left brace has not yet been removed from the input stream. The argument to \a is whatever lies between \a and the first {1 token.
Arguments to macros are of two types: undelimited and delimited. In the parameter text an undelimited argument is of the form #<n> (with <n> an integer from 1 to 9) directly followed by the next #<n+1> parameter or by the {1 (left brace) that starts the replacement text.
Otherwise an argument is delimited.
When determining the arguments to a macro at call time, TeX distinguishes between delimited and undelimited arguments. If it's looking for an undelimited argument, there are two cases: if a {1 token follows, then the argument is the whole sequence of tokens up to the matching }2 token; otherwise the next token is the argument.
If TeX is looking for a delimited argument, it will absorb tokens until finding the exact sequence of tokens specified as delimiter. All tokens so absorbed (excluding the delimiting tokens) form the argument; the delimiting tokens are discarded. During such process, TeX does no interpretation whatsoever of the tokens it absorbs.
Exception, which is directly related to exercise 20.5: if the parameter text is #1# (but it could be more complicated), then the delimiter of the argument #1 (in general of the last argument) is {1 which is not discarded as it would be with standard delimited arguments.
You can think of it like this:
\def\foo#1;{}
will read everything after \foo until there is a ;, so \foo 123; will read 123 as #1 and remove ; from the input stream as well.
Similarly
\def\foo#1#{}
will create a macro that has a right delimited argument, but instead of ; this right delimiter is {, so it reads everything up to the next opening brace, but contrarily to the first case, in this case { will not be removed from the input stream (well, actually it will be removed but reinserted by \foo's replacement text). So \def\foo#1#{} behaves like \def\foo#1;{;} but uses { as the right delimiter instead of ;.
You can see that the opening brace is actually removed and later reinserted by taking a look at the \meaning (or \showing the definition):
\def\foo#1#{}\show\foo
\def\foo#1#{foo}\show\foo
will print
> \foo=macro: #1{->{. > \foo=macro: #1{->foo{.
to the terminal.
EDIT: This tries to answer the comment
As I understand it, the parameter text is a regular expression. In this regular expression the opening parenthesis
{is a delimiter of the replacement text and is therefore removed when replacing. This is not the case when it is immediately preceded by#. What is the purpose of this rule?
If TeX would also remove that opening brace what would be left would be an unbalanced token list (an unmatched closing brace). Therefore if the following macros aren't created very carefully, this would throw an error, and creating macros with the same logic would be much harder. So the only purpose of this rule is that you can create macros which are right delimited by a token of category code 1 (a { in normal catcodes) without needing a lot of macros to sanitize the now unbalanced input stream.
Imagine the following situation:
\def\foo#1#{}
\foo 123{abc}
After the definition of \foo and its expansion what would be left in the input stream if the { wasn't reinserted would be
abc}
and we'd have to somehow sanitize that unmatched closing brace. Say we want to create a macro which reads everything up to an opening brace and the next group, what we'd have to do now would be to create a macro that grabs every token until it meets a closing brace, but \def\bar#1}{} will throw an error as well, so how should we create this? What we'd need to do would be something like the following (note that I create the unbalanced text by expanding an \iffalse{\fi in the following):
\documentclass[]{article}
\makeatletter
\long\def\grabuntilclosingbrace@fi@firstoftwo\fi\@secondoftwo#1#2{\fi#1}
\def\grabuntilclosingbrace
{%
\begingroup
\aftergroup\grabuntilclosingbrace@done
\grabuntilclosingbrace@a
}
\def\grabuntilclosingbrace@a
{%
\futurelet\grabuntilclosingbrace@tok\grabuntilclosingbrace@b
}
\def\grabuntilclosingbrace@b
{%
\ifx\grabuntilclosingbrace@tok\egroup
\grabuntilclosingbrace@fi@firstoftwo
\fi
\@secondoftwo
{%
\afterassignment\grabuntilclosingbrace@final
\let\afterassignment@tok=%
}
{%
\grabuntilclosingbrace@c
}%
}
\def\grabuntilclosingbrace@final
{%
\aftergroup\grabuntilclosingbrace@end
\endgroup
}
\long\def\grabuntilclosingbrace@done#1\grabuntilclosingbrace@end
{Argument was: \texttt{#1}}
\long\def\grabuntilclosingbrace@c#1{\aftergroup#1\grabuntilclosingbrace@a}
\makeatother
\begin{document}
\expandafter\grabuntilclosingbrace\iffalse{\fi abc}
\end{document}
And this macro can't deal with spaces or nested groups. See how complicated life would be, if TeX didn't give us the super handy \def\foo#1#{} rule?
If you want to know, what this rule can be used for:
Say we have some macro that can't deal with nested groups in its argument, so we have to test whether the argument has a group, after all we want to give a helpful error message instead of just letting our macro fail. So we need to create a test that tests for a nested group. With the logic of \def\foo#1#{} we can reduce this to a test whether an argument is empty (this reuses code/ideas from https://tex.stackexchange.com/a/517265/117050).
\documentclass[]{article}
\makeatletter
\long\def\ifgroupin#1%
{%
\ifgroupin@a#1{}\ifgroupin@tokB\ifgroupin@false
\ifgroupin@tokA\ifgroupin@tokB\@firstoftwo
}
\long\def\ifgroupin@a#1#{\ifgroupin@b}
\long\def\ifgroupin@b#1{\ifgroupin@c\ifgroupin@tokA}
\long\def\ifgroupin@c#1\ifgroupin@tokA\ifgroupin@tokB{}
\long\def\ifgroupin@false\ifgroupin@tokA\ifgroupin@tokB\@firstoftwo#1#2{#2}
\makeatother
\begin{document}
\ifgroupin{abc}{true}{false}
\ifgroupin{a{b}c}{true}{false}
\end{document}
An alternative version which uses the classic \if\relax\detokenize{#1}\relax empty test, because this might be easier to understand (but takes about 160% the time of the previous implementation):
\makeatletter
\long\def\ifgroupin#1%
{%
\if\relax
\detokenize\expandafter\expandafter\expandafter{\ifgroupin@a#1{}}\relax
\expandafter\@secondoftwo
\else
\expandafter\@firstoftwo
\fi
}
\long\def\ifgroupin@a#1#{\@gobble}
\makeatother