\DeclareLanguageMappingSuffix, inheritance, and polyglossia in biblatex

Update All multilingual features of biblatex should work as intended with biblatex v3.14 (or above) and polyglossia v1.46 (or above).

The MWE from the question produces the expected (babel-equivalent) output now.

If you are facing any problems with polyglossia and biblatex, update your system so that you have at least biblatex 3.14 (and a matching Biber version) and polyglossia 1.46. If that does not help, please open a bug report at https://github.com/plk/biblatex/issues.

The rest of this answer below is kept for academic purposes.

Note in particular that the bug in polyglossia (https://github.com/reutenauer/polyglossia/issues/205) that I opened when formulating this question is resolved.

This is not a full answer, but it describes why things go wrong with polyglossia and how the entire language mapping business works.

polyglossia and biblatex

The polyglossia support in biblatex is still patchy at best. My advice would be to avoid polyglossia if at all possible and use babel instead. polyglossia used to be a must-have for certain languages (especially for languages with non-Latin scripts or RTL languages), but recent babel development has shown improvements in that matter. For other languages (Western European certainly) there is no huge advantage in choosing polyglossia over babel (and it has been that way for quite some time).

babel's languages vs. polyglossia main language+options

The most problematic issue by far is the detection of language variants. While babel simply and naively defines a new language for each variant or dialect it encounters, polyglossia has a much more sophisticated interface that sets the main language that can then be modified with certain attributes (for example variant to turn english into american or british or german into swissgerman or austrian; and spelling to select ngerman or german).

Variant detection

Unfortunately, the variant/spelling variants of a main language have a non-uniform implementation across languages and there is no interface for packages or classes to find out which variant is currently in use. This is a major problem for packages that also want to act upon the language variant. It is actually worse than that, in certain situations not even polyglossia knows which variant is active! Take the following contrived example.

\documentclass{article}

\usepackage{xcolor}
\usepackage{polyglossia}
\setotherlanguage{english}
\setotherlanguage{dutch}
\setmainlanguage{german}

\def\month{1}% we need january where German German and Austrian German differ
\begin{document}
\tableofcontents

\begin{english}
\section{\protect\today}
\end{english}

\section{\protect\today}

\begin{german}[variant=austrian]
\section{\protect\textcolor{red}{\protect\today}}
\end{german}

\begin{dutch}
\section{\protect\today}
\end{dutch}
\end{document}

The date changes languages in the TOC in the same way that it changes in the section headers, but it does not change the language variant. I have reported this at https://github.com/reutenauer/polyglossia/issues/205

Finding out which language to load

biblatex mainly works with the babel model of treating languages: Every dialect is a new language (there can be inheritance between languages, but that does not really matter for the overall view). So there would first have to be a proper interface to convert polyglossia main languages+options to a babel equivalent before biblatex can even know what language to load. That conversion is a non-trivial task because there need not be a connection between the way polygossia sets up its main language with options and the name babel has for the resulting language. I believe that this can only be properly tackled on the polyglossia side, because each gloss-....ldf essentially has to declare its babel-equivalent names. A very rudimentary proof of concept can be found at https://github.com/moewew/polyglossia/tree/bblnames, the changes to the current polyglossia version are at https://github.com/reutenauer/polyglossia/compare/master...moewew:bblnames.

Using the loaded language

Even after the language file is finally loaded, the problems for biblatex don't stop. With babel the bibstrings are simply added to \extras<babel-language>. That command is called whenever the language is switched and so the correct strings are activated. But with polyglossia only \extras<base-language> exists (there is no \extrasaustrian there is only \extrasgerman), biblatex would have to compute the base language from the babel language, all variant changes would have to be made within \extras<main-language>. But as mentioned above, there is no uniform implementation of the variants, so this requires intimate knowledge of the gloss-....ldfs.

biblatex's language mapping

Since this question is also concerned with language mapping and that changed recently (in version 3.11, see CHANGES.md), let me explain how this works now.

Whenever biblatex now tries to load an .lbx file it tries to resolve the language mapping first.

There are specific language mappings with \DeclareLanguageMapping{<language>}{<new lbx file>} and general mappings for all languages with \DeclareLanguageMappingSuffix{<suffix>} that map <language>.lbx to <language><suffix>.lbx. The specific mapping takes precedence over a general mapping.

.lbx files may inherit from other .lbx files with \InheritBibliographyStrings{<lang>}, \InheritBibliographyExtras{<lang>} or using inherit = {<lang>}, as special 'bibstring' in \DeclareBibliographyStrings. .lbx files that do not define all strings or extras themselves should always inherit what they don't define from a sensible source (see for example Problems with biblatex' \DeclareLanguageMapping after update).

When biblatex resolves the inheritance all language mappings are still applied. Only infinite recursion is avoided.

Assume we have a file german.lbx with all the definitions for German.

The file ngerman.lbx for new orthography rules inherits from german and only changes what is necessary, it looks like this (simplified and shortened)

\ProvidesFile{ngerman.lbx}

\InheritBibliographyExtras{german}

\DeclareBibliographyStrings{%
  inherit  = {german},
  citedas  = {{im Folgenden zitiert als}{im Folgenden zit\adddotspace als}},
}

We also have two files for \DeclareLanguageMappingSuffix{-dp}

german-dp.lbx

\ProvidesFile{german-dp.lbx}

\InheritBibliographyExtras{german}

\DeclareBibliographyStrings{%
  inherit  = {german},
  castle   = {{Schloß}{Schloß}},
}

ngerman-dp.lbx

\ProvidesFile{ngerman-dp.lbx}

\InheritBibliographyExtras{ngerman}

\DeclareBibliographyStrings{%
  inherit  = {ngerman},
  castle   = {{Schloss}{Schloss}},
}

If you now ask biblatex to get you ngerman, it will recall the language mapping we set up and open ngerman-dp.lbx instead. ngerman-dp.lbx inherits from ngerman and so ngerman.lbx is opened (we don't follow the mapping because we don't want to open ngerman-dp.lbx and land in an infinite loop). ngerman.lbx in term inherits from german, this time the mapping is followed and so german-dp.lbx is opened. german-dp.lbx inherits from german and finally german.lbx is read. The following list shows the files opened in reverse chronological order, since inherit is always the first keyword definitions are overwritten by subsequent files

• german.lbx
• german-dp.lbx
• ngerman.lbx
• ngerman-dp.lbx

For ngerman you will end up with

citedas  = {{im Folgenden zitiert als}{im Folgenden zit\adddotspace als}},% ngerman.lbx
castle   = {{Schloss}{Schloss}},% ngerman-dp.lbx

For a weird example assume now that additionally to the above

• german.lbx defines lipsum = {lorem}
• ngerman.lbx defines lipsum = {ipsum}
• german-dp.lbx defines lipsum = {dolor}
• ngerman-dp.lbx has no entry for lipsum.

What do we end up with for ngerman (which by virtue of the mapping is ngerman-dp.lbx)? The lists shows that we get 'ipsum', i.e. the definition from ngerman.lbx.

I should add that I don't have a strong inclination towards claiming that this chain of inheritance is the correct or at least expected one. But I do feel that the way to set up the .lbx files is natural and I so I am prepared to accept the resulting chain of inheritance as natural corollary.

If ngerman-dp.lbx inherited from german-dp instead we would get the following chain

• german.lbx (inherited from german-dp.lbx)
• german-dp.lbx (inherited from ngerman-dp.lbx)
• german-dp-dp.lbx (ngerman-dp.lbx inherits german-dp, this is the result of applying the suffix, the file doesn't exist and is ignored after a warning, instead german-dp.lbx is loaded)
• ngerman-dp.lbx

In particular no definition of ngerman.lbx gets read and german-dp.lbx 's lipsum = {dolor} wins. We end up with

citedas  = {{im folgenden zitiert als}{im folgenden zit\adddotspace als}},% german.lbx
castle   = {{Schloss}{Schloss}},% ngerman-dp.lbx
lipsum   = {{dolor}{dolor}},% german-dp.lbx

ngerman-dp.lbx would have to repeat all definitions from ngerman.lbx that it wants to keep.

Imho the most important thing when setting up language files is to get the inheritance chain correct. You want to inherit in american-test from american and from english-test but it is unclear which from both should be the direct parent: The file says american but the expected output shows that you want english-test to win at least for the pages field.

I would setup the files so that english-test is the direct parent along with a mapping to overcome the deficiencies of polyglossia.

\documentclass{article}

\usepackage{filecontents}
\begin{filecontents}{english-test.lbx}
\InheritBibliographyExtras{english}
\DeclareBibliographyStrings{%
  inherit = {english},
  in = {in \textcolor{red}{(english)}}{in \textcolor{red}{(english)}},
  pages = {pages \textcolor{red}{(english)}}{pp\adddot\space\textcolor{red}{(english)}}
}
\end{filecontents}
\begin{filecontents}{american-test.lbx}
\InheritBibliographyExtras{english-test}
\DeclareBibliographyStrings{%
  inherit = {english-test},
  in = {in \textcolor{blue}{(american)}}{in \textcolor{blue}{(american)}},
}
\end{filecontents}

\usepackage{xcolor}
\usepackage{csquotes}
\usepackage{polyglossia}
\setmainlanguage[variant=american]{english}

%\usepackage[american]{babel}% <- works with babel
\usepackage[style=verbose]{biblatex}
\DeclareLanguageMappingSuffix{-test}
 \DeclareLanguageMapping{english}{american-test}
\addbibresource{biblatex-examples.bib}

\begin{document}
\cite{westfahl:space}
\end{document}