Chemistry - Why a TS must have only one negative frequency
Solution 1:
Well, let's just look at the simple case you mention. If we are at a transition state, then we are at a saddle point on the potential energy surface (PES). As you say, there are two directions where we go down in energy and two where we go up in energy which are along or perpendicular to the gradient vector.
Note that to confirm we are at a saddle point, in mathematics one could do the second-derivative test. Or, in chemistry (and in math) in order to do this, we construct the Hessian matrix and find its determinant. So, let our PES be given by a function $f(x,y)$. Then, the Hessian takes the form, $$ \begin{bmatrix} f_{xx} & f_{xy} \\ f_{xy} & f_{yy} \\ \end{bmatrix} $$ where $f_{xx}$ denotes the second partial derivative of $f$, both with respect to $x$ and likewise for the rest of the terms. Notice that $f_{xy}=f_{yx}$, so this matrix is symmetric and hence all its eigenvalues are real. Also note that sometimes people will write this in terms of the force constants instead of the second-derivatives because a harmonic force-constant is a second derivative.
So, if we take the determinant of this matrix, its sign tells us about whether the function is concave up or concave down at the point we are interested in. This determinant is given by, $$ f_{xx}f_{yy}-(f_{xy})^2. $$ The sign of this actually tells us information about the eigenvalues of the matrix because the determinant of a matrix is equal to the product of its eigenvalues. Notice that in order to find the eigenvalues of this matrix we would solve the following equation: $$ (f_{xx}-\lambda)(f_{yy}-\lambda)-(f_{xy})^2=0 $$ Or, after rearranging, $$ \lambda^2-\lambda(f_{xx}+f_{yy})+f_{xx}f_{yy}-f_{xy}^2=0 $$
These eigenvalues can be related to the frequencies of the vibrations if we first mass-weighted each second-derivative. All I want to show here is that for this case there are only two eigenvalues, and these eigenvalues do not have to do with the number of directions to move towards a minimum. One of the vibrations you can imagine as being the mode which in two directions have an increase in energy. So, along this mode, the molecule stays at the transition state and hence there is a physical frequency associated with this motion. If the second derivative test is worth anything, then the overall sign of this determinant will be negative at a transition state and hence there will be one and only one negative eigenvalue. Note that this only works at critical points, so that's why at the transition state there must be only one negative imaginary frequency (or negative eigenvalue).
Because for most every molecule we have more than two modes to worry about, you could imagine splitting a much larger matrix into many 2-d matrices and looking at two modes at a time. From this idea it is clear that a transition state need not be limited to having only one negative frequency. Rather, at second-order saddle points, (i.e. a saddle point where following the eigenvector corresponding to a negative eigenvalue leads you to another saddle point) there will be two negative imaginary eigenvalues.
To summarize:
- The number of imaginary frequencies is the number of negative eigenvalues of the Hessian matrix (the matrix of all possible mixed-second-partial derivatives)
- There can be more than one negative frequency at a higher-order saddle point, but physically speaking these are much less common than transition states with only one imaginary frequency
- One cannot determine the number of imaginary frequencies by thinking about the number of paths to a minimum. You could think about the number of modes which do not kick you off of a saddle point if you prefer this.
Edit:
I mistakenly referred to negative frequencies which is not correct (as alluded to by a commenter). The eigenvalues of this matrix correspond to the force constants (second derivatives with respect to displacement coordinates), so one gets negative force constants, not negative frequencies at a saddle point. Because the force constant for one mode is negative at a transition state, the "frequency" of this mode is evaluated as $\nu=\sqrt{\frac{k}{\mu}}$ as usual, and the frequency is imaginary. So, one is looking for either one negative force constant or one imaginary frequency, not a negative frequency. Sorry I missed that.
Solution 2:
I will answer from a practical point of view. First off, let's note that a transition state (TS) is, loosely speaking, in-between a reactant and a product (state). These constitute local minima, possibly with very low barriers. Moving along an appropriately defined reaction coordinate, you go from the reactant through the TS to the product or vice versa (think reaction with equilibrium). So having "two directions", that are actually just one path on which you are going forward or backward, makes sense.
In terms of performing calculations, one finds that it can be difficult to obtain even minima that do not have spurious negative frequencies. The dreaded methyl rotations are the most common example. (One implication is that free enthalpies, which take into account the vibrational component of the partition function, may easily be wrong in cases of methyl rotations.) For a TS, this is even more difficult, because the algorithms used for geometry optimization are less robust than those for minima. However, as noted in the comments to your question, a second-order saddle point (meaning two negative frequencies) is often accompanied by a first-order saddle point close by, which is usually a better TS.
Note however, that this is not a strict requirement. Depending on the chemical species involved, degenerate TS have been observed, essentially existing where two pathways cross. (I do not have a reference on hand.) These are very rare and I assume require a lot of symmetry. So for standard chemistry, the rule of "TS means stationary point and exactly one negative frequency" holds.
Further note that in order to be sure about the TS being connected to the product and reactant you have in mind, one should modify the TS geometry to follow the pseudo-vibrational mode twice (once for each direction) and optimize for a minimum. Some programs even have automation for this. This procedure leads to the reactant and product of that TS.