How to suppress a "Rare" ligature that pre-empts a "Common" ligature for the same character pair?

Some thing like this using feature files should work:

First the feature file:

# scripts and languages, if the font use others they should be defined here too
languagesystem DFLT dflt;
languagesystem latn dflt;

# this undoes the effect of rare ligature on ft
feature liga {
  sub f_t.alt by f t;
} liga;

# move rare ft to historic ligatures
feature hlig {
  sub f t by f_t.alt;
} hlig;

# apply the regular ligature on ft and fft
feature liga {
  sub f f t by f_f_t;
  sub f t by f_t;
} liga;

The order of the above is important, this way regular “ft” and “fft” are used even with Rare ligatures, but not with Historic.

The glyph names should match those in the font with f_t.alt being the name of the unwanted “ft” ligature, and f_t & f_f_t being the names of the wanted ones.

The full feature file syntax is documented by Adobe (but LuaTeX supports v1.6 f the syntax + FontForge extension, not the current one, so there are some differences).

Then it can be applied to the font by using FeatureFile fontspec option:

\documentclass{article}
\usepackage{fontspec}

\begin{document}
\fontspec[Ligatures={Common,Rare},FeatureFile=test.fea]{Garamond Premier Pro}
ft fft \em ft fft

\fontspec[Ligatures={Common,Rare,Historic},FeatureFile=test.fea]{Garamond Premier Pro}
ft fft \em ft fft
\end{document}

My answer is essentially the same as that given by @svenper but it’s too long to be a comment and it may address Mico’s questions to svenper.

Perhaps I have a more recent version — version 2.000 — of Garamond Premier Pro than you, because with Ligatures=Rare, I get the rare “ft” and the common “fft” in both the upright and the italic. (Some foundries notify customers of updates or even send free updates; Adobe doesn’t, as if its reputation were enhanced by leaving lesser versions of its products to roam the wild.)

If you’re stuck with a defective version of the fonts and dislike their discretionary “ft” ligature (it does nothing for legibility), and if you want a convenient way to use all discretionary ligatures except for the “ft,” you can define f\_t.alt into oblivion:

\documentclass{article}
\usepackage{fontspec}
\directlua{
  fonts.handlers.otf.addfeature{
    name = "mlig",
    type = "substitution",
    data = {
      ["f_t.alt"] = "f_t",
    },
  }
}
\setmainfont{Garamond Premier Pro}[
  Ligatures=Rare,% placing this before the new feature may be important; please test
  RawFeature=+mlig]% mlig = “Mico’s ligatures”
\begin{document}
ft fft ct sp st

\em ft fft as at ct et ij is ll sp st ta th us
\end{document}

If you want to restore the discretionary “ft” in one spot, simply use {\addfontfeatures{RawFeature=-mlig}ft}.

Please test this; it works for me, but then my copy of the fonts doesn’t show the original problem.

Another approach is to refrain from specifying Ligatures=Rare, and instead to define a new feature with only th discretionary ligatures you like. For example, suppose you like many of the discretionary ligatures but not “ct,” “sp,” and “st.” In that case, the common ligatures suffice for the upright fonts, but you need a new feature for the italics — let’s call it ilig, for “italic ligatures.”

\documentclass{article}
\usepackage{fontspec}
\directlua{
  fonts.handlers.otf.addfeature{
    name = "ilig",
    type = "ligature",
    data = {
      ['a_s'] = { "a", "s" },
      ['a_t'] = { "a", "t" },
      ['e_t'] = { "e", "t" },
      ['i_j'] = { "i", "j" },
      ['i_s'] = { "i", "s" },
      ['l_l'] = { "l", "l" },
      ['t_a'] = { "t", "a" },
      ['t_h'] = { "t", "h" },
      ['u_s'] = { "u", "s" },
    },
  }
}
\setmainfont{Garamond Premier Pro}[
  ItalicFeatures={RawFeature=+ilig}]
\begin{document}
ft fft ct sp st

\em ft fft as at ct et ij is ll sp st ta th us
\end{document}

Finally, suppose you need to avoid dlig because of its effect on the f-ligatures, but you want the upright and italic “ct,” “sp,” and “st” ligatures, along with a selection of the discretionary italic ligatures. You can simply proceed as above with ilig, and also add the discretionary ligatures found in both upright and italic to liga, which is already on by default.

\documentclass{article}
\usepackage{fontspec}
\directlua{
  fonts.handlers.otf.addfeature{
    name = "liga",
    type = "ligature",
    data = {
      ['c_t'] = { "c", "t" },
      ['s_p'] = { "s", "p" },
      ['s_t'] = { "s", "t" },
    },
  }
  fonts.handlers.otf.addfeature{
    name = "ilig",
    type = "ligature",
    data = {
      ['a_s'] = { "a", "s" },
      ['a_t'] = { "a", "t" },
      ['e_t'] = { "e", "t" },
      ['i_j'] = { "i", "j" },
      ['i_s'] = { "i", "s" },
      ['l_l'] = { "l", "l" },
      ['t_a'] = { "t", "a" },
      ['t_h'] = { "t", "h" },
      ['u_s'] = { "u", "s" },
    },
  }
}
\setmainfont{Garamond Premier Pro}[
  ItalicFeatures={RawFeature=+ilig}]
\begin{document}
Th γγ λλ ffi ffj ffl ft fft ct sp st

\em Th γγ λλ ffi ffj ffl ft fft as at ct et ij is ll sp st ta th us
\end{document}

(Don’t try adding ligatures found only in the italic to liga. With my newer version of Garamond Premier Pro and an up-to-date TeX Live 2017, TeX seems forgiving, but I’ve played with typefaces in which TeX looked for, say, l_l in the upright font and, not finding it, produced “aegory” where I wrote “allegory.”)

With recent versions of TeX Live feature files are deprecated. With LuaLaTeX, another way is to change the glyph of the ligature.

\directlua{
    fonts.handlers.otf.addfeature {
        name = "norareft",
        type = "substitution",
        data = {
            ["f_t.alt"] = "f_t",
        }
    }
}

And then add the fontspec option RawFeature=+norareft.

---

A full MWE:

\documentclass{article}
\directlua{
    fonts.handlers.otf.addfeature {
        name = "norareft",
        type = "substitution",
        data = {
            ["f_t.alt"] = "f_t",
        }
    }
}
\usepackage{fontspec}
\setmainfont{GaramondPremrPro}% 
        [RawFeature=+norareft,   % <-- the new feature
         Ligatures={Common,Rare},% enable both common and rare ligs        
         Extension     = .otf,
         ItalicFont    = GaramondPremrPro-It,
         BoldFont      = GaramondPremrPro-Smbd,
         BoldItalicFont= GaramondPremrPro-SmbdIt]

\begin{document}
ft fft \textit{ft fft}

\bfseries ft fft \textit{ft fft}
\end{document}