How can I prove that a state of equilibrium is unstable?

In the centre of a bowl there is equilibrium.

• Put a ping pong ball in it. If this ball is ever shaken slightly away from equilibrium, it will immediately roll back. A "shake-proof" equilibrium is called stable.

• Now, turn the bowl around and put the ball on the top. This is an equilibrium. But at the slightest shake, the ball rolls down. This "non-shake-proof" equilibrium is called unstable.

It is about the potential energy. Because, systems always tend towards lowest potential energy. The bottom of the bowl is of lowest potential energy, so the ball wants to move back when it is slightly displaced. The top of the flipped bowl is of highest potential energy, and any neighbour point is of lower energy. So the ball has no tendency to roll back up.

Mathematically, it is thus all about figuring out if the equilibrium is a minimum or a maximum. Only a minimum is stable.

You might for many practical/physical purposes be able to determine this by simply looking at the graph of the potential energy.

But mathematically, this can be solved directly from the potential energy expression $U$. Just look at the sign of the double derivative (derived to position).

• If it is positive, $U''_{xx}>0$, then the value at the equilibrium is about to increase - so it is a minimum.
• If it is negative, $U''_{xx}<0$, then the value at the equilibrium is about to decrease - so it is a maximum.

If you have a 2D function, then you have more than one double derivative, $U''_{xx}$, $U''_{xy}$, $U''_{yx}$ and $U''_{yy}$. In this case, you must collect them into a so-called Hessian matrix and look at the eigenvalues of that matrix. If both positive, then the point is a minimum; if both negative, then the point is a maximum. (And if a mix, then the point is neither a minimum nor a maximum, but a saddle point).

This may be a bit more than you expected - but it is the rather elegant, mathematical method.

Is it enough to state that for any none-null coordinates, the electric field isn't zero, ergo the equilibrium is unstable?

No, that is not enough.
You are right with: At the point of equilibrium the electric force needs to be null.
But furthermore: The direction of the electric force in the surroundings of the equilibrium position is important.

• If the electric force points towards the equilibrium position, then the equilibrium is stable.
• If the electric force points away from the equilibrium position, then the equilibrium is instable.

Or is there a more elegant way of proving it?

It is usually easier to analyze equilibrium with potential energy, instead of with forces.

• If the potential energy is a minimum, then the equilibrium is stable.
• If the potential energy is a maximum, then the equilibrium is instable.

If there is no charge at the origin (that is producing the electric field), then by Gauss's law (in derivative form), the divergence of the electric field there is 0: $\frac{\partial ^2 V}{\partial x^2} + \frac{\partial ^2 V}{\partial y^2} + \frac{\partial ^2 V}{\partial z^2}= 0$. Unless all three of these quantities are zero, then one of them must be negative, which means that in that direction, the equilibrium is unstable.