Why does $\epsilon$ come first in the $\epsilon-\delta$ definition of limit?

Good question!

The answer is that if we put $\delta$ first, then our definition would no longer correspond to what we mean by the limit of a function.

Proposition Let $f$ be the constant function $0$. Let $c,L$ be any real numbers. Then:

For all $\delta>0$ there exists $\epsilon>0$ such that if $0<|x-c|<\delta$, then $|f(x)-L|<\epsilon$.

In other words, if we define $\lim'$ by putting the $\delta$ first, then $\lim_{x\to c}'f(x)=L$.

Proof. Let $\delta>0$ and let $\epsilon=|L|+1$. If $0<|x-c|<\delta$ then $$|f(x)-L| = |0-L|=|L|<|L|+1=\epsilon\quad\Box$$

So our constant function $0$, which should clearly converge to $0$ at every point, under the new definition converges to every real number at every point.

In particular, limits aren't unique and the properties we expect to hold for a 'limit' no longer hold. It just isn't a very interesting definition and it no longer captures our intuition of what limits should be.

By contrast, the definition with $\epsilon$ first exactly captures the notion of a 'limit'.

As John Gowers explained in his answer, if you reverse the order of quantification, you can say that $\lim_{x \to 0} x = \pi$ or $\lim_{x \to 0} x = -1$. The modified definition is clearly not that of limit. Let's use different notation and write $\text{mil}_{x \to c} f(x) = L$ for the "backwards" definition.

What have we defined? If there exist $c$ and $L$ in $\mathbb{R}$ such that $\text{mil}_{x \to c} f(x) = L$, it means that for all $c' \in \mathbb{R}$, $\lim_{x \to c'} f(x) \not\in \{\infty, -\infty\}$.

In fact, if $f$ diverges for some $c' \in \mathbb{R}$, and $\delta$ is chosen large enough, so that $|c'-c| < \delta$, then we cannot find $\epsilon$ such that $|f(x)-L| < \epsilon$ for all $x$ such that $|x-c| < \delta$. On the other hand, if $f$ does not go to infinity for finite argument, we can always pick a suitably large $\epsilon$.

We can further note that if there exist $c$ and $L$ in $\mathbb{R}$ such that $\text{mil}_{x \to c} f(x) = L$, then for all $c'$ and $L'$ in $\mathbb{R}$, it is also true that $\text{mil}_{x \to c'} f(x) = L'$. This tells us that there are better ways to define finite-valued functions. Let's not go down that route. Instead, note that in the previous paragraph, I used the word "large" twice.

If we try to explain the definition of $\lim$, we may say, "for every $\epsilon$, however small, there's a sufficiently small $\delta$." The definition of $\text{mil}$, however, would be read, "for every $\delta$, no matter how large, there is a sufficiently large $\epsilon$."

Clearly, neither "small" nor "large" appears in the definitions. However, when we try to elucidate one of these definitions, we intuitively think of it as defining the rules of a two-player game. Let the two players be Adam ($\forall$dam) and Eve ($\exists$ve).

Eve chooses the values of the existentially quantified variables; Adam chooses the values of the universally quantified variables. Eve tries to make the claim true; Adam tries to make it false. If Adam picks $\epsilon$ first, his way to make it hard for Eve to respond is to choose a small $\epsilon$. In fact, if Adam wins by choosing $\epsilon_0$, he also wins with any $\epsilon$ such that $0 < \epsilon \leq \epsilon_0$.

However, if Adam chooses $\delta$ first, he's better off choosing a large $\delta$. Similar considerations apply to Eve's choices. Bottom line: giving $\delta$ to Adam and $\epsilon$ to Eve results in a very different game; one that does not define limits.

Changing the order of the picks also changes the game substantially. Asking Eve to go first, and pick $\delta$ before Adam picks $\epsilon$ puts her at a great disadvantage. (Try it.) The resulting game, once again, does not define limits.

It is also possible for a change in the order of the picks to result in two different games that define related, yet distinct notions. This is the case of uniform continuity vs. continuity. Winning the uniform continuity game is harder for Eve, but the class of functions on which she can win is an interesting one, and both definitions are therefore in use.