Formally étale at all primes does not imply formally étale?

Using the module of Kähler differentials, it is easy to show that $R\to S$ is formally unramified if and only if the induced maps $R\to S_{\mathfrak{p}}$ are formally unramified for all primes $\mathfrak{p}\subset S$.

Consider a presentation of $S$ over $R$ as $R[X]/I$ in generators and relations, where $R[X]:=R[X_m]_{m\in M}$ is a polynomial ring in a possibly infinite family of indeterminates indexed by $M$, and $I\subset R[X]$ is an ideal. Fix a family of generators of $I=(F_j)_{j\in J}$ indexed by $J$, again not necessarily finite.

It is enough to show that $R\to S$ is formally smooth. This is equivalent to showing that there exists a morphism of $R$-algebras that is a splitting for the canonical projection $\pi:R[X]/I^2 \to R[X]/I=S$, which will necessarily be unique because $R\to S$ is formally unramified.

Let $\overline{X}_m$ denote the image of $X_m$ in $R[X]/I^2$. We must find elements $\delta_m\in I/I^2$ such that $(\forall j\in J)F_j(X_m + \delta_m)=0$. We rewrite this using Taylor's formula as $$\bar{F}_j+ \sum_{m\in M}\overline{\frac{\partial F_j}{\partial X_m}}\delta_m=0.$$

Rearranging, we get a system of equations indexed by $J$ $$(*)_{j\in J} \qquad \sum_{m\in M}\overline{\frac{\partial F_j}{\partial X_m}}\delta_m=-\overline{F}_j.$$

We wish to find a unique solution for this system in the $\delta_m$. Since $\Omega_{S/R}=0$, each $dX_m\in \Omega_{R[X]/R}$ is an $S$-linear combination $dX_m=s_{m,1}dF_{j_{m,1}}+\cdots + s_{m,h_m}dF_{j_{m,h_m}}$. If we use the $s_{m,k}$ as coefficients to form $S$-linear combinations of the equations $(*)_{j_k}$, for each $m$, we get an equation of the form $$(**)_m \qquad \delta_m=-(s_{m,1}\overline{F}_{j_{m,1}}+\cdots + s_{m,h_m}\overline{F}_{j_{m,h_m}}).$$

Showing that these define solutions for all of the equations $(*)_j$ is not immediate, but it is a local question on $S$. However, our local rings $S_{\mathfrak{p}}$ are all formally étale, so the local conditions are satisfied. Then this proves the global claim.

(Note: This is not my proof. I've paraphrased the proof communicated to me by Mel Hochster.)

Edit: Fixed LaTeX using Scott's suggestion.

EDIT: Don't bother reading my partial solution. Brian Conrad pointed out that an easier way to do what I did is to use the equivalent definition of formally unramified in terms of Kähler differentials. And later on, fpqc posted below a complete solution passed on by Mel Hochster, who got it from Luc Illusie, who got it from ???.

OLD ANSWER: Here is a half-answer. I'll prove half the conclusion, but on the plus side I'll use only half the hypothesis! Namely, I will prove that if $R \to S_{\mathfrak{p}}$ is formally unramified for all primes $\mathfrak{p} \subset S$, then $R \to S$ is formally unramified.

Let $A$ be an $R$-algebra, and let $I \subseteq A$ be a nilpotent ideal. Given $R$-algebra homomorphisms $f,g \colon S \to A$ that become equal when composed with $A \to A/I$, we must prove that $f=g$. Fix $\mathfrak{p}\subset S$. Then the localizations $A_{\mathfrak{p}} := S_{\mathfrak{p}} \otimes_{S,f} A$ and $S_{\mathfrak{p}} \otimes_{S,g} A$ of $A$ (defined viewing $A$ as an $S$-algebra in the two different ways) are naturally isomorphic, since adjoining the inverse of an $a \in A$ to $A$ automatically makes $a+\epsilon$ invertible for any nilpotent $\epsilon$ (use the geometric series). Now $f$ and $g$ induce $R$-algebra homomorphisms $f_{\mathfrak{p}},g_{\mathfrak{p}} \colon S_{\mathfrak{p}} \to A_{\mathfrak{p}}$ that become equal when we compose with $A_{\mathfrak{p}} \to A_{\mathfrak{p}}/I A_{\mathfrak{p}}$. Since $R \to S_{\mathfrak{p}}$ is formally unramified, this means that $f_{\mathfrak{p}} = g_{\mathfrak{p}}$. In other words, for every $s \in S$, the difference $f(s)-g(s)$ maps to zero in $A_{\mathfrak{p}}$ for every $\mathfrak{p}$. An element in an $S$-module that becomes $0$ after localizing at each prime ideal of $S$ is $0$, so $f(s)=g(s)$ for all $s$. So $f=g$.