Why do we think there are only three generations of fundamental particles?

There are very good experimental limits on light neutrinos that have the same electroweak couplings as the neutrinos in the first 3 generations from the measured width of the $Z$ boson. Here light means $m_\nu < m_Z/2$. Note this does not involve direct detection of neutrinos, it is an indirect measurement based on the calculation of the $Z$ width given the number of light neutrinos. Here's the PDG citation:

http://pdg.lbl.gov/2010/listings/rpp2010-list-number-neutrino-types.pdf

There is also a cosmological bound on the number of neutrino generations coming from production of Helium during big-bang nucleosynthesis. This is discussed in "The Early Universe" by Kolb and Turner although I am sure there are now more up to date reviews. This bound is around 3 or 4.

There is no direct relationship between quark and neutrino masses, although you can derive possible relations by embedding the Standard Model in various GUTS such as those based on $SO(10)$ or $E_6$. The most straightforward explanation in such models of why neutrinos are light is called the see-saw mechanism

http://en.wikipedia.org/wiki/Seesaw_mechanism

and leads to neutrinos masses $m_\nu \sim m_q^2/M$ where $M$ is some large mass scale on the order of $10^{11} ~GeV$ associated with the vacuum expectation value of some Higgs field that plays a role in breaking the GUT symmetry down to $SU(3) \times SU(2) \times U(1)$. If the same mechanism is at play for additional generations one would expect the neutrinos to be lighter than $M_Z$ even if the quarks are quite heavy. Also, as you mentioned, if you try to make fourth or higher generations very heavy you have to increase the Yukawa coupling to the point that you are outside the range of perturbation theory. These are rough theoretical explanations and the full story is much more complicated but the combination of the excellent experimental limits, cosmological bounds and theoretical expectations makes most people skeptical of further generations. Sorry this wasn't mathier.

My research involves a geometric model of spin-1/2 particles, though the discussion of the three generations is beyond the scope of my thesis. However, if I can figure out how to mention this speculation in the Future Work section at the end of my thesis, I will probably do so.

I can't help but marvel at the coincidence of the number three for generations as well as for dimensions of space (where the inertial reference frame fixes the time dimension related to the spatial dimensions). If spin was treated as an oscillation (not just an "intrinsic angular momentum"), then higher-generational particles could have more complicated modes of oscillation: second- and third-generation particles could have two- and three- dimensional spin modes, respectively. If spin was somehow related to mass (which the magnetic dipole moment seems to say it is), then the greater masses of the higher-generational particles could be explained by these higher-dimensional oscillations. Somehow. :)

I am only putting this idea out because I don't suspect I will have the chance to investigate it myself in a more thorough manner. But who knows, maybe I will, and maybe your comments on the idea will help me hone it. Or maybe someone else will take it and run with it, which is fine with me as long as I am mentioned in the credits somewhere. ;)

One part of the answer to this question is that the neutrinos are Majorana particles (or Weyl--- the two are the same in 4d), which can only acquire mass from nonrenormalizable corrections. The neutrinos do not have a right handed partner in an accessible energy range. If there is such a partner, it is very very heavy. So this means that they have to be exactly massless if the standard model is exactly renormalizable.

The interactions that give neutrinos mass are two-Higgs two-Lepton scattering events in the standard model Lagrangian, where the term is $HHLL$ with the SU(2) indices of each H contracted with an L. This term gives neutrino masses, but is dimension 5, so is suppressed by the natural energy scale, which is 10¹⁶ GeV, the GUT scale. This gives the measured neutrino masses. This term also rules out a low energy Planck scale.

If you have another generation, the next neutrino would have to be light, just because of this suppression. There is no way to couple the Higgs to the next neutrino much stronger than the other three. There are only 3 light neutrinos, as revealed by the Z width, BBN, as others said.