Pandoc and foreign characters

By default, Pandoc use the pdflatex engine when converting markdown file to pdf files. pdflatex can not handle Unicode characters very smoothly as xelatex. You should try xelatex instead. But, merely using xelatex command is not enough. As is often the case, you need to choose a proper font which contains glyphs for the Unicode characters your want to typeset.

I am a Chinese user, so take Chinese for example. If you have a test.md which contains the following content:

你好汉字

you can use the following command to compile this markdown file:

pandoc --pdf-engine=xelatex -V CJKmainfont="KaiTi" test.md -o test.pdf

In the above command, --pdf-engine=xelatex is used to select the LaTeX engine (for the new version of Pandoc, --latex-engine option is deprecated). -V CJKmainfont="KaiTi" is used to select the proper font which support Chinese. For other languages, you may use the flag -C mainfont="<FONT_NAME>".

How to find a font which support your language

In order to find a font which supports your language, you need to know your language code. Then, if you are on Linux system or on Windows systems with TeX Live installed. You can use the following command to find a valid font for you language:

fc-list :lang=zh #find the font which support Chinese (language code is `zh`)

The output on my Linux system is shown belowenter image description here

If you choose to use, e.g. the font Source Han Serif CN, then use the following command to compile your markdown file:

 pandoc --pdf-engine=xelatex -V CJKmainfont="Source Han Serif CN" test.md -o test.pdf

UPDATE: the answer below seems to be valid for pandoc 1.x but with later versions the syntax has changed

Coming back to this post in five years time and the issue is still there. The command

pandoc -s test.md -t latex -o test.pdf

fails when test.md contains text with non-latin characters, Greek, Cyrillic, CJK, Hebrew and Arabic included.

LaTeX was designed before Unicode and its support for different character sets is robust in some areas but far from comprehensive, so the advice to use XeLaTeX is valid yet requires one to choose the main font carefully, since there is no automatic choice.

Below is a small taxonomy of possible issues and some solutions. All tested with Pandoc 1.19.

Cyrillic

Support for Cyrillic alphabet in LaTeX is provided via T2A font encoding.

Consider a small sample:

# Header

## Subheader

Tetris (Russian: Тетрис) quoting Wikipedia is a tile-matching puzzle 
video game

Running this example with pandoc would fail with:

! Package inputenc Error: Unicode char Т (U+422)
(inputenc)                not set up for use with LaTeX.

See the inputenc package documentation for explanation.

A fix is available as fontenc option is a predefined variable in default.latex template.

Running this example with

pandoc -t latex -o tetris.pdf -V fontenc=T2A cyrillic.md

would produce correct rendering

Text with cyrillic characters rendered correctly

This however would not handle other language features correctly such as hyphenation. A better way would be to use Babel and have it select the correct font encoding.

pandoc -t latex -o tetris.pdf -V lang -V babel-lang=russian cyrillic.md

Or to switch languages with Babel commands inside Markdown

# Header

## Subheader

Tetris (Russian: \foreignlanguage{russian}{Тетрис}) quoting Wikipedia 
is a tile-matching puzzle video game

And run with

pandoc -t latex -o tetris.pdf -V lang -V babel-lang=english \
       -V babel-otherlangs=russian cyrillic2.md

Greek

The example in the original post contains characters both from the main and extended Greek Unicode codepages.

Anyway, the widely used LGR greek font encoding is not covered by LaTeX 3 project and is classified as a local encoding, i.e. it may vary from site to site and from system to system according to the LaTeX Encoding Guide.

On TeX Live the following packages need to be installed: texlive-greek-inputenc, texlive-greek-fontenc and texlive-cbfonts. Note that you need Babel 3.9 or later. However the result of

pandoc -t latex -o anarchy.pdf -V fontenc=LGR greek.md

may appear unexpected.

Text with both Greek and Latin characters typed as Greek

In order to correct this issue one has to setup LaTeX Babel package correctly. And insert commands to switch between the languages in the original text:

# Header!

## Sub Header

themselves derived respectively from the Greek \textgreek{ἀναρχία} 
i.e. 'anarchy'

Compiling this with the following command

pandoc -s greek2.md -t latex -V fontenc=T2A -V lang -V babel-lang=english \
    -V babel-otherlangs=greek -o greek.pdf

would produce the output exactly as you would expect it to be:

Text with greek characters rendered correctly

XeLaTeX

All of this would not be needed if we were using XeLaTeX.

Just running the original example with 

pandoc -s greek.md --latex-engine=xelatex -t latex -o greek.pdf

would produce

Text with Greek characters omitted

Because the font does not contain anything in the greek character positions the output contains some white space instead.

Selecting one of the popular fonts as the new mainfont would help a bit

pandoc -s greek.md --latex-engine=xelatex \
    -V mainfont="Liberation Serif" -t latex -o greek.pdf

Text with only basic Greek characters rendered correctly

However characters from the extended Greek codepage such as the small letter alpha with psili accent are not rendered.

The Font Setup for Greek with XeTeX/LuaTeX Guide suggests to use DejaVu, Libertine or Free font families.

Indeed with DejaVu Serif, Linux Libertine O as well as Tempora and perhaps some other fonts, the result would be as expected. See below the rendering with XeLaTeX and Linux Libertine fonts.

pandoc -s greek.md --latex-engine=xelatex -V mainfont="Linux Libertine O" \
      -t latex -o greek.pdf

Text with Greek characters rendered correctly with XeLaTeX and Libertine fonts


Use the --pdf-engine=xelatex option.


Works for Cyrillic characters

pandoc myfile.md --pdf-engine=xelatex -V mainfont=Arial

Tags:

Pdf

Markdown

Pandoc

Recent Posts