How to know whether an Ordinary Differential Equation is Chaotic?

No general tools are known.

For the Lorenz equations, Warwick Tucker has proved the existence of a chaotic strange attractor.

It is a non-trivial proof that combines normal form theory and validated computations using interval arithmetic.

An introduction to the Lorenz system can be found in [1,2].

If there is no general tool to prove that a continuous dynamical system is chaotic, there are at least several tools to prove that a system is not chaotic (see e.g. [3]). Here is a short non-exhaustive list of features which allow a first-order autonomous ODE system $$ \dot{X} = F(X) \, \qquad\text{where}\qquad X\in\mathbb{R}^n \;\text{and}\; F \in C^1 $$ to be chaotic:

• $F$ is nonlinear
• the phase space dimension $n$ is equal to $3$ or larger (consequence of the Poincaré-Bendixson theorem)
• there is at least one eigenvalue of the Jacobian matrix $\partial F/\partial X$ evaluated at the equilibria of the system that has a non-negative real part (consequence of the Hartman-Grobman theorem)

There are several case-dependent methods for the analysis of chaos. In the case of periodically forced Hamiltonian systems, a dedicated tool is Melnikov's method.

To answer the question at hand: Provided that we decide to agree on a particular definition of chaotic system, it might be possible to determine the chaotic nature of a system from its equations (not sure if there is a general method).

My definition of chaotic system: Given a system of (ordinary) differential equations $\Delta$, we say the system is chaotic if there exists initial data, $d_0$, such that we can find a point $p_0$ along solution curve for the given initial data, which arrises from the independent variables taking on the value $x$ ($x$ is a point in the space defined by the independent variables of $\Delta$) and an $\epsilon >0$, for which there is no $\delta >0$ satisfying the following:

Given initial data $d$ contained in the $\delta$-ball centered at $d_0$, the point $p$ corresponding to the independent variables taking on the value $x$ is within the $\epsilon$-ball centered at $p_0$.

The point of the above definition is that infinitesimal changes in intial data yield arbitrary changes in what the solution curve looks like (you can see intuitively how your given system fails to have a $\delta$ by observing what happens if you take an $\epsilon$-ball of a point on one of the 'wings' and have the initial data be from where the 'wings' meet).