Characterization of sets of differentiability

(Expanded from my comments, with next day corrections incorporated into my initial answer)

NOTATION: Let $C(f)$ and $\Delta(f)$ be the sets of points of continuity and finite differentiability, respectively, of the function $f.$

Theorem: $\;{\mathbb R} - \Delta(f) \;$ is $\;G_{\delta \sigma}$

First, Henning Makholm's accepted answer to Continuous functions are differentiable on a measurable set? shows that $\Delta(f)$ is $G_{\delta \sigma \delta},$ as you correctly stated.

For completeness, note that Henning Makholm wrote

$$\forall\varepsilon:\exists \delta:\exists Y:\forall h: |h| < \delta \Rightarrow |F(x,h) - Y| < \varepsilon$$ $[\ldots]$

For each particular choice of $\varepsilon$, $\delta$, $Y$, and $h$, the set of $x$ such that $|h| < \delta\Rightarrow |F(x,h)-Y| < \varepsilon$ is open $[\ldots]$

Thus, we get

$$\begin{array} {} & {\forall h} & {} & {\exists \delta \; \exists Y} & {} & {\forall \varepsilon} & {} \\ {\text{open}} & {\longrightarrow} & {G_{\delta}} & {\longrightarrow} & {G_{\delta \sigma}} & {\longrightarrow} & {G_{\delta \sigma \delta}} \\ {} & {\cap} & {} & {\cup \cup} & {} & {\cap} & {} \\ \end{array} $$

Therefore, $\Delta(f)$ is $G_{\delta \sigma \delta}.$ Hence, by taking the complement, we see that ${\mathbb R} - \Delta(f)$ is $F_{\sigma \delta \sigma}$ (by standard De Morgan's Laws computations).

However, we can reduce the complexity by an entire Borel class by arranging things so that we start with closed sets. Specifically, by replacing $|F(x,h)-Y| < \varepsilon$ with $|F(x,h)-Y| \leq \varepsilon$ (which does not change the set of points involved), we get $$\forall\varepsilon:\exists \delta:\exists Y:\forall h: |h| < \delta \Rightarrow |F(x,h) - Y| \leq \varepsilon$$ which leads to $$\begin{array} {} & {\forall h} & {} & {\exists \delta \; \exists Y} & {} & {\forall \varepsilon} & {} \\ {\text{closed}} & {\longrightarrow} & {\text{closed}} & {\longrightarrow} & {F_{\sigma}} & {\longrightarrow} & {F_{\sigma \delta}} \\ {} & {\cap} & {} & {\cup \cup} & {} & {\cap} & {} \\ \end{array} $$ This allows us to strengthen the result: The set of points of finite differentiability is $F_{\sigma \delta},$ and hence ${\mathbb R} - \Delta(f)$ is $G_{\delta \sigma}.$

Converse Results

From my answer to Continuous functions are differentiable on a measurable set? it follows that any $G_{\delta}$ set can be a ${\mathbb R} - \Delta(f)$ set, and it follows that any Lebesgue measure zero $G_{\delta \sigma}$ set can be a ${\mathbb R} - \Delta(f)$ set. More generally, the union of any two such sets can be a ${\mathbb R} - \Delta(f)$ set, plus all these converses can be obtained by using continuous functions. I'm nearly certain that some positive measure $G_{\delta \sigma}$ sets cannot be a ${\mathbb R} - \Delta(f)$ set (even if we use discontinuous functions), but at the moment I do not have a reference or an example to offer.

(ADDED NEXT DAY) I looked through some papers and other items at home this morning, and the chronologically ordered list of references below consists of a few of the more relevant items I found. At some later time I might revisit this thread and provide descriptions of what is proved and/or stated in the papers (except the lengthy survey papers), but for now I'll post the list without descriptions for others who might be interested. For example, Zahorski (1941, 1946) proves the characterization of ${\mathbb R} - \Delta(f)$ sets when $f$ is continuous, and Brudno (1943) extends this to arbitrary functions, with the additional condition that there exists a $G_{\delta}$ set lying between (subset relation) the smaller set ${\mathbb R} - C(f)$ (set of points of discontinuity) and the larger set ${\mathbb R} - \Delta(f)$ (set of points of non-finite differentiability). I'll mention in passing that Brudno's result includes the often re-discovered fact (see the end of [14] here) that if the set of discontinuity points of a function is dense, then the set of non-finite differentiability points of the function is co-meager (complement of a first Baire category set).

An Answer (with proof) to Ian's Question

The original question was whether there exists a $G_{\delta \sigma}$ set that is not a ${\mathbb R} - \Delta(f)$ set. It turns out that Zahorski answers this question at the end of his 1946 paper (and probably also at the end of his 1941 paper, but I can't read Russian). In fact, Zahorski (on p. 178 of his 1946 paper) even shows that there exist $F_{\sigma}$ sets that are not ${\mathbb R} - \Delta(f)$ sets, which is a much stronger result. (The $G_{\delta \sigma}$ sets include all $F_{\sigma}$ sets, and many other sets.) Here is Zahorski's argument, substantially expanded.

Let $E \subseteq {\mathbb R}$ be $F_{\sigma}$ and first Baire category in ${\mathbb R}$ with the property that $E \cap I$ has positive Lebesgue measure for every open interval $I.$ One way to obtain such a set is choose a closed nowhere dense set with positive measure from each bounded open interval in $\mathbb R$ with rational endpoints, and then take the union of these countably many closed nowhere dense sets. I claim that $E$ cannot be written as the union of a $G_{\delta}$ set and a measure zero $G_{\delta \sigma}$ set, and hence $E$ cannot be a ${\mathbb R} - \Delta(f)$ set. For a later contradiction, suppose $E = H \cup Z,$ where $H$ is a $G_{\delta}$ set and $Z$ has measure zero. (I won't even need to use the fact that $Z$ is a $G_{\delta \sigma}$ set.) Then $H$ has a positive measure intersection with every open interval (because $E$ has this property and $Z$ has measure zero). A very weak consequence of this is that $H$ has a nonempty intersection with every open interval, and hence $H$ is dense in ${\mathbb R}.$ Thus, $H$ is a dense $G_{\delta}$ subset of ${\mathbb R},$ which implies that $H$ is not a first category subset of ${\mathbb R},$ and now we have a contradiction because we initially assumed that $H$ was a first category set.

Chronological List of References

Zygmunt Zahorski, О множестве точек недиференцируемости непрерывной функции [On the nondifferentiability sets of continuous functions], Matematiceskii Sbornik (N.S.) 9(51) #3 (1941), 487-510.

A. Brudno, Непрерывность и дифференцируемость [Continuity and differentiability], Matematiceskii Sbornik (N.S.) 13(55) #1 (1943), 119-134.

Zygmunt Zahorski, Sur l'ensemble des points de non-dérivabilité d'une fonction continue [On the set of points of non-differentiability of a continuous function], Bulletin de la Société Mathématique de France 74 (1946), 147-178.

E. M. Landis, On the set of points of existence of an infinite derivative (Russian), Doklady Akademii Nauk SSSR [= Comptes Rendus de l'Académie des Sciences de l'URSS] (N.S.) 107 (1956), 202-204.

V. M. Tsodyks [Tzodiks, Codyks], On sets of points where the derivative is correspondingly finite or infinite (Russian), Doklady Akademii Nauk SSSR [= Comptes Rendus de l'Académie des Sciences de l'URSS] (N.S.) 113 (1957), 36-38.

V. M. Tsodyks [Tzodiks, Codyks], On sets of points where the derivative is correspondingly finite or infinite (Russian), Doklady Akademii Nauk SSSR [= Comptes Rendus de l'Académie des Sciences de l'URSS] (N.S.) 114 (1957), 1174-1176.

V. M. Tsodyks [Tzodiks, Codyks], О множествах точек, где производная равна соответственно $+\infty$ и $-\infty$ [On sets of points where the derivative is respectively equal to $+\infty$ or $-\infty$], Matematiceskii Sbornik (N.S.) 43(85) #4 (1957), 429-450.

Solomon Marcus, Points of discontinuity and points of differentiability (Russian), Académie de la République Populaire Roumaine. Revue de Mathématiques Pures et Appliquées [after 1963: Revue Roumaine de Mathématiques Pures et Appliquées] 2 (1957), 471-474.

Solomon Marcus, Sur les propriétés différentielles des fonctions dont les points de continuité forment un ensemble frontière partout dense [On the differentiability properties of functions whose points of continuity form a dense boundary set], Annales scientifiques de l'École Normale Supérieure (3) 79 #1 (1962), 1-21.

Andrew Michael Bruckner and John Lander Leonard, Derivatives, American Mathematical Monthly 73 #4 (April 1966) [Part II: Papers in Analysis, Herbert Ellsworth Slaught Memorial Papers #11], 24-56. [See 2nd half of p. 39.]

George Piranian, The set of nondifferentiability of a continuous function, American Mathematical Monthly 73 #4 (April 1966) [Part II: Papers in Analysis, Herbert Ellsworth Slaught Memorial Papers #11], 57-61.

L. I. Kaplan, The mutual position of sets where the derivative is finite and infinite, Siberian Mathematical Journal 18 #4 (July-August 1977), 570-581.

L. I. Kaplan, Points of discontinuity of a function, and points where an infinite derivative exists, Russian Mathematical Surveys 35 #6 (1980), 97-98.

L. I. Kaplan, Points of continuity of a function and points where there is a finite and an infinite derivative, Russian Mathematical Surveys 36 #5 (1981), 155-156.

L. I. Kaplan, Points of continuity of a function and points of existence of finite and infinite derivative, Siberian Mathematical Journal 24 #6 (Nov.-Dec. 1983), 876-889.

Bogusław Kaczmarski, On the set of points at which a function has no one-sided derivative, Demonstratio Mathematica 18 #4 (1985), 1127-1141.

Bogusław Kaczmarski, On the category and Borel type of the set of points of one-sided non-differentiability, Demonstratio Mathematica 22 #2 (1989), 441-460.

Bogusław Kaczmarski, On the measure and Borel type of the set of points of one-sided non-differentiability, Demonstratio Mathematica 23 #1 (1990), 267-270.

Bogusław Kaczmarski, The sets where a function has infinite one-sided derivatives, Real Analysis Exchange 16 #2 (1990-1991), 421-424.

Bogusław Kaczmarski, On the sets where a continuous function has infinite one-sided derivatives, Real Analysis Exchange 23 #1 (1997-1998), 343-356.