Chemistry - Why is it wrong to use the concept of hybridization for transition metal complexes?
Solution 1:
Tetrahedral complexes
Let's consider, for example, a tetrahedral $\ce{Ni(II)}$ complex ($\mathrm{d^8}$), like $\ce{[NiCl4]^2-}$. According to hybridisation theory, the central nickel ion has $\mathrm{sp^3}$ hybridisation, the four $\mathrm{sp^3}$-type orbitals are filled by electrons from the chloride ligands, and the $\mathrm{3d}$ orbitals are not involved in bonding.
Already there are several problems with this interpretation. The most obvious is that the $\mathrm{3d}$ orbitals are very much involved in (covalent) bonding: a cursory glance at a MO diagram will show that this is the case. If they were not involved in bonding at all, they should remain degenerate, which is obviously untrue; and even if you bring in crystal field theory (CFT) to say that there is an ionic interaction, it is still not sufficient.
If accuracy is desired, the complex can only really be described by a full MO diagram. One might ask why we should believe the MO diagram over the hybridisation picture. The answer is that there is a wealth of experimental evidence, especially electronic spectroscopy ($\mathrm{d-d^*}$ transitions being the most obvious example), and magnetic properties, that is in accordance with the MO picture and not the hybridisation one. It is simply impossible to explain many of these phenomena using this $\mathrm{sp^3}$ model.
Lastly, hybridisation alone cannot explain whether a complex should be tetrahedral ($\ce{[NiCl4]^2-}$) or square planar ($\ce{[Ni(CN)4]^2-}$, or $\ce{[PtCl4]^2-}$). Generally the effect of the ligand, for example, is explained using the spectrochemical series. However, hybridisation cannot account for the position of ligands in the spectrochemical series! To do so you would need to bring in MO theory.
Octahedral complexes
Moving on to $\ce{Ni(II)}$ octahedral complexes, like $\ce{[Ni(H2O)6]^2+}$, the typical explanation is that there is $\mathrm{sp^3d^2}$ hybridisation. But all the $\mathrm{3d}$ orbitals are already populated, so where do the two $\mathrm{d}$ orbitals come from? The $\mathrm{4d}$ set, I suppose.
The points raised above for tetrahedral case above still apply here. However, here we have something even more criminal: the involvement of $\mathrm{4d}$ orbitals in bonding. This is simply not plausible, as these orbitals are energetically inaccessible. On top of that, it is unrealistic to expect that electrons will be donated into the $\mathrm{4d}$ orbitals when there are vacant holes in the $\mathrm{3d}$ orbitals.
For octahedral complexes where there is the possibility for high- and low-spin forms (e.g., $\mathrm{d^5}$ $\ce{Fe^3+}$ complexes), hybridisation theory becomes even more misleading:
Hybridisation theory implies that there is a fundamental difference in the orbitals involved in metal-ligand bonding for the high- and low-spin complexes. However, this is simply not true (again, an MO diagram will illustrate this point). And the notion of $\mathrm{4d}$ orbitals being involved in bonding is no more realistic than it was in the last case, which is to say, utterly unrealistic. In this situation, one also has the added issue that hybridisation theory provides no way of predicting whether a complex is high- or low-spin, as this again depends on the spectrochemical series.
Summary
Hybridisation theory, when applied to transition metals, is both incorrect and inadequate.
It is incorrect in the sense that it uses completely implausible ideas ($\mathrm{3d}$ metals using $\mathrm{4d}$ orbitals in bonding) as a basis for describing the metal complexes. That alone should cast doubt on the entire idea of using hybridisation for the $\mathrm{3d}$ transition metals.
However, it is also inadequate in that it does not explain the rich chemistry of the transition metals and their complexes, be it their geometries, spectra, reactivities, or magnetic properties. This prevents it from being useful even as a predictive model.
What about other chemical species?
You mentioned that hybridisation works well for "other compounds." That is really not always the case, though. For simple compounds like water, etc. there are already issues associated with the standard VSEPR/hybridisation theory. Superficially, the $\mathrm{sp^3}$ hybridisation of oxygen is consistent with the observed bent structure, but that's just about all that can be explained. The photoelectron spectrum of water shows very clearly that the two lone pairs on oxygen are inequivalent, and the MO diagram of water backs this up. Apart from that, hybridisation has absolutely no way of explaining the structures of boranes; Wade's rules do a much better job with the delocalised bonding.
And these are just Period 2 elements - when you go into the chemistry of the heavier elements, hybridisation generally becomes less and less useful a concept. For example, hypervalency is a huge problem: $\ce{SF6}$ is claimed to be $\mathrm{sp^3d^2}$ hybridised, but in fact $\mathrm{d}$-orbital involvement in bonding is negligible. On the other hand, non-hypervalent compounds, such as $\ce{H2S}$, are probably best described as unhybridised - what happened to the theory that worked so well for $\ce{H2O}$? It just isn't applicable here, for reasons beyond the scope of this post.
There is probably one scenario in which it is really useful, and that is when describing organic compounds. The reason for this is because tetravalent carbon tends to conform to the simple categories of $\mathrm{sp}^n$ $(n \in \{1, 2, 3\})$; we don't have the same teething issues with $\mathrm{d}$-orbitals that have been discussed above. But there are caveats. For example, it is important to recognise that it is not atoms that are hybridised, but rather orbitals: for example, each carbon in cyclopropane uses $\mathrm{sp^5}$ orbitals for the $\ce{C-C}$ bonds and $\mathrm{sp^2}$ orbitals for the $\ce{C-H}$ bonds.
The bottom line is that every model that we use in chemistry has a range of validity, and we should be careful not to use a model in a context where it is not valid. Hybridisation theory is not valid in the context of transition metal complexes, and should not be used as a means of explaining their structure, bonding, and properties.
Solution 2:
For the most part, coordination bonds involve d electrons from the transition metal interacting with some other electrons on a separate molecule. Hybrid orbitals are constructed from a single atom's orbitals mixing. If you do hybridization for the metal, then you are mixing its s, p, and d orbitals together in attempt to explain geometry. But ultimately, it is incorrect as really only the d orbitals interact. That's why people get upset when you use hybridization theory for coordination bonds.
Solution 3:
Hybridization theory states that methane's four sigma bonds are equal (sp3).
However, if you apply group theory to the four sigma bonds (use the character table for Td point-group), the irreducible representation, if memory serves, = A1 + T2 (A1=single and T2 = a triple degeneracy).
H.T. is a nice tool to help understand introductory molecular geometry and/or o-chem, but, as it is superceded by M.O. theory, it has absolutely no basis in reality.
There is a very good description of a square planar complex above. The simple answer to your question: Almost every d8 transition metal complex is square-planar.
Regarding the rest (this also depends upon the d electron count and is a generalization ), Oh is more favorable than Td, and Td is more favorable than SP. Again, take this with a grain of salt, but it is a decent starting point. Consider the #of ligands and the # of occupied co-ordination sites on the TM center resulting from a L occupying a single co-ordination site--then remember that if a ligand can occupy multiple CN sites, it typically will do so. Regarding the above 'favorability generalisation', remember that lower CN complexes are less stable and, thus, more open to attack because of unoccupied CN sites--think about a medieval town vs a walled medieval town; more L or more occupied co-ordination sites= bigger wall (our TM friend wants to have the largest wall allowed by the system). More often than not, your textbook's and/or prof's examples will rely upon a strict adherence to the 18 electron rule, so keep this in mind when trying to determine whether or not the addition of a MM bond and/or consideration of a dimer is apt (see below).
You should also consider whether or not a M-M bond is favorable (this is typically encountered with odd # d TM complexes--this is not an absolute) because then you would need to consider whether or not dimerization has occured (remember though that poly-dentate ligands may also result in the formation of multiple metal-center complexes that lack MM bonds). H.T. completely fails here--this is why you should learn how to determine a molecule's point-group and ignore H.T.