How do I get a token list with an end-of-line character in it?
This seems to work
{\char_set_catcode_other:N \^^M\tl_gset:Nn\g_tmpa_tl{^^M}}
With the help of \tl_analysis_show:n
you can get a better result:
\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\cs_generate_variant:Nn \tl_replace_all:Nnn {NV}
\str_const:Nx \c_other_eol_str { \cs_to_str:N \^^M }
\NewDocumentCommand \getstuff {+v}
{
\tl_analysis_show:n { #1 } % for debugging
\tl_set:Nn \l_tmpa_tl {#1}
\tl_replace_all:NVn \l_tmpa_tl \c_other_eol_str {!wibble!}
\tl_analysis_show:N \l_tmpa_tl % for debugging
\tl_use:N \l_tmpa_tl
}
\ExplSyntaxOff
\begin{document}
\getstuff+
some stuff
+
\getstuff+some stuff+
\end{document}
The log has, for the first call of \getstuff
,
The token list contains the tokens:
> ^^M (the character ^^M)
> s (the character s)
> o (the character o)
> m (the character m)
> e (the character e)
> (blank space )
> s (the character s)
> t (the character t)
> u (the character u)
> f (the character f)
> f (the character f)
> ^^M (the character ^^M).
This confirms that ^^M
has been absorbed as a token with category code 12; since \c_other_eol_str
contains exactly a token with the same property, the next diagnostic message says
The token list \l_tmpa_tl contains the tokens:
> ! (the character !)
> w (the letter w)
> i (the letter i)
> b (the letter b)
> b (the letter b)
> l (the letter l)
> e (the letter e)
> ! (the character !)
> s (the character s)
> o (the character o)
> m (the character m)
> e (the character e)
> (blank space )
> s (the character s)
> t (the character t)
> u (the character u)
> f (the character f)
> f (the character f)
> ! (the character !)
> w (the letter w)
> i (the letter i)
> b (the letter b)
> b (the letter b)
> l (the letter l)
> e (the letter e)
> ! (the character !).
Hurray! The ^^M
has been correctly recognized and changed into !wibble!
.
A different strategy is by using l3regex
:
\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewDocumentCommand \getstuff {+v}
{
\tl_analysis_show:n { #1 } % for debugging
\tl_set:Nn \l_tmpa_tl {#1}
\regex_replace_all:nnN { \r } { !wibble! } \l_tmpa_tl
\tl_analysis_show:N \l_tmpa_tl % for debugging
\tl_use:N \l_tmpa_tl
}
\ExplSyntaxOff
\begin{document}
\getstuff+
some stuff
+
\getstuff+some stuff+
\end{document}
In this case !wibble!
will be all with category code 12 characters in the modified token list; but of course you don't really want to change ^^M
into !wibble!
, do you?