How to handle verbatim material in LuaLaTeX?

Rather than re-invent the wheel, I prefer to refer to and build on Paul Isambert's wonderful article LuaTEX: What it takes to make a paragraph, published in 2011 in Volume 32 of TUGBoat.

In this article, the author discusses inter alia how one can make use of the LuaTeX's process_input_buffer callback to handle verbatim material in a very robust and flexible manner. To quote from the article:

One of the most arcane areas of TeX is catcode management. This becomes most important when one wants to print verbatim text, i.e. code that TeX should read as characters to be typeset only, with no special characters, and things turn definitely dirty when one wants to typeset a piece of code and execute it too (one generally has to use an external file). With the process_input_buffer callback, those limitations vanish: the lines we would normally pass to TeX can be stored and used in various ways afterward.

The MWE shown below adapts Paul's TUGBoat article code to LaTeX coding practices. (Paul's code is distinctly PlainTEX-ish.) If you're at all interested in this issue, you should absolutely read Paul's original article; the passage on handling verbatim material is on pages 2 and 3 of the pdf file version of the article (pp. 69 and 70 in original published journal volume). Paul explains the issues far better than I possibly could.

  • The Lua-side code consists of defining a Lua table (which will hold the verbatim material) and 3 Lua functions that do most of the work.

  • The TeX-side code sets up several TeX macros that activate, invoke, and suspend the operation of the Lua functions. The main LaTeX macros are \printverbatim (you should be able to guess what it does...) and \useverbatim; the latter executes whatever is in the most recently read verbatim chunk.

  • A verbatim chunk is anything sandwiched between lines that say \Verbatim and \EndVerbatim, respectively. If there's any material on the lines that contain the strings \Verbatim and \EndVerbatim, respectively, it will be ignored.

Note that I have not defined a LaTeX environment called Verbatim, although it wouldn't be too hard to do so.

enter image description here

\documentclass{scrreprt}
%% Set up a Lua table and 3 Lua functions
\directlua{%
verb_table = {}
function store_lines (str)
  if string.find (str , "\noexpand\\EndVerbatim" ) then
    luatexbase.remove_from_callback (
      "process_input_buffer" , "store_lines")
  else
    table.insert(verb_table, str)
  end
  return ""
end
function register_verbatim ()
  verb_table = {}
  luatexbase.add_to_callback(
    "process_input_buffer" , store_lines , "store_lines")
end
function print_lines ( catcode )
  if catcode then
    tex.print ( catcode , verb_table)
  else
    tex.print ( verb_table )
  end
end
}
%% TeX-side code: define several macros
\def\Verbatim{\directlua{register_verbatim()}}
\def\useverbatim{\directlua{print_lines()}}
\def\printverbatim{%
  \par
  \bgroup
  \setlength{\parindent}{0pt}
  \ttfamily
  \directlua{print_lines(1)}
  \egroup
}
\def\createcatcodes{%
  \bgroup
  \catcode`\\=12 \catcode`\{=12 \catcode`\}=12
  \catcode`\$=12 \catcode`\&=12 \catcode`\^^M=13
  \catcode`\#=12 \catcode`\^=12 \catcode`\_=12
  \catcode`\ =13 \catcode`\~=12 \catcode`\%=12
  \savecatcodetable 1 % '\savecatcodetable' is a LuaTeX primitive
  \egroup}
\createcatcodes
\def\Space{ }
\bgroup
\catcode`\^^M=13\gdef^^M{\quitvmode\par}%
\catcode`\ = 13\gdef {\quitvmode\Space}%
\egroup

\begin{document}

\Verbatim
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\EndVerbatim
\printverbatim

\bigskip
\Verbatim
\def\lululu{%
Lua\kern-.01em
\TeX
}%
\EndVerbatim
\printverbatim

\bigskip
\useverbatim % this "registers" the macro "\lululu"
\noindent\lululu
\end{document}

Mico's answer is very good and shows how a low-level implementation of verbatim buffers in LuaTeX could look like.

ConTeXt comes with this feature included and it is super easy to use.

\starttext

\startbuffer[terminal]
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\stopbuffer

\typebuffer[terminal]

\startbuffer[logo]
\def\lululu{%
Lua\kern-.01em
\TeX
}%
\stopbuffer

\typebuffer[logo]

\getbuffer[logo]
\lululu

\stoptext

The output is similar to Mico's answer.


An alternative, that tries to imitate ConTeXt buffers is the scontents package. The output obtained is the same as in Mico's response, of course it is not "bullet proof", but, allow me to customize the verbatim output using fancyvrb and tcolorbox as I read in your comments.

The main function of the package is to store content, but try to imitate \typebuffer using verbatim, anyway, here's my answer:

\documentclass{article}
\usepackage{scontents}
\setlength{\parindent}{0pt}
\pagestyle{empty}
\begin{document}
\section{default}
% stored in terminal
\begin{scontents}[store-env=terminal]
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\end{scontents}
\typestored[1]{terminal}

% stored in logo
\begin{scontents}[store-env=logo]
\def\lululu{
Lua\kern-.01em
\TeX
}
\end{scontents}
\typestored[1]{logo}
\getstored[1]{logo}
\lululu
\end{document}

output-default

Customizing the output of \typestoredusing fancyvrb and tcolorbox:

\documentclass{article}
\usepackage{scontents}
\makeatletter
\let\verbatimsc\@undefined
\let\endverbatimsc\@undefined
\makeatother
\usepackage{fancyvrb,tcolorbox}
\newenvironment{verbatimsc} %
{\VerbatimEnvironment
\begin{tcolorbox}[colback=gray!25, boxsep=0pt, arc=0pt, boxrule=0pt]
\begin{Verbatim}[fontsize=\small]} %
{\end{Verbatim} %
\end{tcolorbox}}
\setlength{\parindent}{0pt}
\pagestyle{empty}
\begin{document}
\section{Using tcolorbox and fancyvrb}
% stored in terminal
\begin{scontents}[store-env=terminal]
total 10244
drwxr-xr-x 47 User Users   16384 Mar 31 18:26 .
dr-xr-xr-x  8 User Users    4096 Jun 27  2015 ..
drwxr-xr-x  9 User Users    4096 Dec 23 12:49 .editor
-rw-r--r--  1 User Users    2950 Mar 31 09:01 .bash_history
\end{scontents}
\typestored[1]{terminal}

% stored in logo
\begin{scontents}[store-env=logo]
\def\lululu{
Lua\kern-.01em
\TeX
}
\end{scontents}
\typestored[1]{logo}
\getstored[1]{logo}
\lululu

\end{document}

output-custom