What are some examples of narrowly missed discoveries in the history of mathematics?
Emil L. Post was very close to proving Gödel's incompleteness theorem, and the existence of algorithmically unsolvable problems in the early 1920s. He realized that one could enumerate all algorithms, and hence obtain an unsolvable problem by diagonalization. Moreover, the "problem" can be viewed as a computable list of questions $Q_1,Q_2,Q_3,\ldots$ for which the sequence of answers (yes or no) is not computable. It follows that there cannot be any complete formal system that proves all true sentences of the form "The answer to $Q_i$ is yes" or "The answer to $Q_i$ is no," because this would solve the unsolvable problem.
But then Post was stuck because he needed to formalize the notion of computation. He had in fact (an equivalent of) the right definition, but logicians were not ready for a definition of computation, and did not believe there was such a thing until the Turing machine concept came along in 1936. Gödel avoided this problem when he proved his theorem (1930) by proving incompleteness of a particular system (Principia Mathematica).
Freeman Dyson discusses a few examples of this in his article Missed Opportunities. One that I thought was particularly striking was that mathematicians could have discovered special relativity decades before Einstein just by staring at Maxwell's equations hard enough, and also on the basis that the representation theory of the Poincare group is simpler than the representation theory of the Galilean group.
In the book The Scientists by John Gribbin, he mentions that, in his search for the theory of general relativity, Einstein apparently wrote down a correct equation that would have led him to correctly discovering the rest of the equations for general relativity very quickly. But, he did not see the equation for what it was and ran down the wrong path for two entire years before coming back to the correct equation. Here's the quote from the book:
"Einstein himself is often presented as the prime example of someone who did great things alone, without the need for a community. This myth was fostered, perhaps even deliberately, by those who have conspired to shape our memory of him. Many of us were told a story of a man who invented general relativity out of his own head, as an act of pure individual creation, serene in his contemplation of the absolute as the First World War raged around him.
It is a wonderful story, and it has inspired generations of us to wander with unkempt hair and no socks around shrines like Princeton and Cambridge, imagining that if we focus our thoughts on the right question we could be the next great scientific icon. But this is far from what happened. Recently my partner and I were lucky enough to be shown pages from the actual notebook in which Einstein invented general relativity, while it was being prepared for publication by a group of historians working in Berlin. As working physicists it was clear to us right away what was happening: the man was confused and lost - very lost. But he was also a very good physicist (though not, of course, in the sense of the mythical saint who could perceive truth directly). In that notebook we could see a very good physicist exercising the same skills and strategies, the mastery of which made Richard Feynman such a great physicist. Einstein knew what to do when he was lost: open his notebook and attempt some calculation that might shed some light on the problem.
So we turned the pages with anticipation. But still he gets nowhere. What does a good physicist do then? He talks with his friends. All of a sudden a name is scrawled on the page: 'Grossman!!!' It seems that his friend has told Einstein about something called the curvature tensor. This is the mathematical structure that Einstein had been seeking, and is now understood to be the key to relativity theory.
Actually I was rather pleased to see that Einstein had not been able to invent the curvature tensor on his own. Some of the books from which I had learned relativity had seemed to imply that any competent student should be able to derive the curvature tensor given the principles Einstein was working with. At the time I had had my doubts, and it was reassuring to see that the only person who had ever actually faced the problem without being able to look up the answer had not been able to solve it. Einstein had to ask a friend who knew the right mathematics.
The textbooks go on to say that once one understand the curvature tensor, one is very close to Einstein's theory of gravity. The questions Einstein is asking should lead him to invent the theory in half a page. There are only two steps to take, and one can see from this notebook that Einstein has all the ingredients. But could he do it? Apparently not. He starts out promisingly, then he makes a mistake. To explain why his mistake is not a mistake he invents a very clever argument. With falling hearts, we, reading the notebook, recognize his argument as one that was held up to us as an example of how not to think about the problem. As good students of the subject we know that the agument being used by Einstein is not only wrong but absurd, but no one told us it was Einstein himself who invented it. By the end of the notebook he has convinced himself of the truth of a theory that we, with more experience of this kind of stuff than he or anyone could have had at the time, can see is not even mathematically consistent. Still, he convinced himself and several others of its promise, and for the next two years they pursued this wrong theory. Actually the right equation was written down, almost accidentally, on one page of the notebook we looked at it. But Einstein failed to recognize it for what it was, and only after following a false trail for two years did he find his way back to it. When he did, it was questions his good friends asked him that finally made him see where he had gone wrong."