Why is "totally ordered" necessary in this implication of the Axiom of Foundation

Take set $\{0,\{1\}\}$. It is not totally ordered under $\in$ because neither $0\in\{1\}$ nor $\{1\}\in 0$, and thus not well-ordered under $\in$. Still, it satisfies Axiom of Foundation.

This comes down to the way "well-ordering" is defined.

Any non-empty set $S$ is well-founded - a well-founded partial order is one in which every (nonempty) set has a (possibly not unique) minimal element. A well-order, by contrast, is a particular kind of linear order; and $S$ need not be linearly ordered by $\in$!

Here's a way to rephrase the bolded statement:

Foundation says that the $\in$-relation on any set $S$ is well-founded; and so, in case $S$ is linearly ordered by $\in$, we will have that $\in$ is a well-ordering on $S$ (since the well-founded linear orders are exactly the well-orderings).