Groups that do not exist

There was a point during the history of the Classification when pursuers of sporadic groups distinguished the Baby Monster, the Middle Monster and the Super Monster. The first two actually turned out to exist (though the word "Middle" was dropped), but the third turned out to be a dud.

http://www.neverendingbooks.org/index.php/tag/simples/page/2 has an account of this.

I'm not sure if this is quite what you're looking for but....

In this book "Finite simple groups", Gorenstein tells the story of Feit & Thompson's proof of the odd order theorem. Very roughly, it goes as follows:

Suppose $G$ is a simple group of odd order. Thompson studied the local structure of the group $G$ to obtain information about the structure of the maximal subgroups of $G$. Feit then applied the Brauer-Suzuki theory of exceptional characters to derive a great deal of character-theoretic information about the group $G$. So far so good.

But now they hit a problem. They were seeking, of course, to demonstrate a contradiction. But, as Gorenstein tells it, one of the possible configurations of maximal subgroups & character information proved extremely difficult to disprove. In the spirit of this question, one might say they found an example of a "group that does not exist". In the end, after spending a year being stuck, Thompson managed to demonstrate the required contradiction by a very delicate analysis of the generators and relations of the putative group $G$.

(I don't have a copy of Gorenstein's book with me. If I get chance I might return to this answer so I can provide some quotes. Gorenstein's account of the whole enterprise is really terrific.)

I tried to write a longer answer which froze, so I'll write a shorter version. You might look at the history of the "Solomon fusion system" which arose in a characterization problem undertaken by Ron Solomon in his work on the classification of finite simple groups. This does not occur in a finite group, but was shown by Dave Benson to occur in a group like topological object ( "2-adic loop space") called BDI(4).

In some sense this led to work by topologists (especially Broto, Levi and Oliver) on "$p$-local finite groups" (actually topological spaces, not groups) which need to associate a linking system to a fusion system of a finite $p$-group. Aschbacher and Chermak showed in an Annals paper a few years ago that the Solomon fusion system does have an associated linking system, an therefore there is a $2$-local finite group associated to that fusion system. More recently, Chermak has shown that there is a $p$-local finite group associated to every saturated fusion system on a finite $p$-group.