Why cannot a set be its own element?

We cannot admit that there exists a set whose members are all the topological spaces. That will lead to a logical contradiction, that there will be a set who is a member of itself.

This is not quite untrue, but at the very least, its a deceptive statement. Sure, in the usual formulation of set theory (namely, ZFC), no set can be a member of itself, but this is basically because (to oversimplify a little) we declare via axiom that "there are no infinite descending membership chains" (see also, axiom of regularity). This means that, in particular, if there was a set $x$ such that $x \in x$, then we'd have

$$\cdots x \in x \in x$$

which is an infinite decreasing membership chain. So, that would be a contradiction. The point, though, is that there are perfectly good variants of ZFC in which there exist sets $x$ satisfying $x \in x$, see also non-well-founded set theory.

Indeed, topology can be developed pretty much independently of whether or not well-founded sets exist. Therefore, that quote is not an example of good mathematical writing.

However, the author is correct that there cannot be a set of all topological spaces. Now you may say:

Wait, I can define the notion of a topological space. Its a pair $(X,\tau)$ such that [whatever]. And if I can define a concept, then the set of all instances of that concept must exist. So, the set of all topological spaces must exist.

Sounds convincing, right? Indeed for quite a long time, mathematicians believed the basic idea that "if you can define a concept, then the set of all instances that concept must exist." It was Russell whom first realized that this principle, which is formalized by the (contradictory) axiom schema of unrestricted comprehension, is untenable. The proof goes something like this. Suppose that for any formula $\phi(x)$ I can write down, there is a set of all $x$ satisfying $\phi(x)$. Then, there is a set of all $x$ satisfying $x \notin x$, call it $R$. Thus $y \in R$ iff $y \notin y$, for all $y$. So $R \in R$ iff $R \notin R$, a contradiction. Basically, this happens because we cannot consistently decide whether or not $R$ should be an element of itself. See also, Russell's Paradox.

So, we need some new set-existence principles, which (hopefully!) don't lead to an outright contradiction like that. The standard collection of set-existence principles is called ZFC. What you've got to understand about ZFC is, its based on the philosophy that "the only reason a set shouldn't exist, is if its too large." So for example, we have an axiom schema (namely, replacement) stating that, if the domain of a (definable) function exists, then its range (aka image) must also exist. This makes sense in light of the philosophy of ZFC, because the range of a function always has cardinality less-than-or-equal-to the cardinality of its domain. So, using this principle, we can always use a set $X$ to prove the existence of a lot of smaller sets (as well as different sets of equal size).

Returning to the issue of topological spaces, it turns out that ZFC cannot prove the existence of a set of all topological spaces. Indeed, ZFC can prove that there is no set of all topological spaces, because such a set would be "too big." The problem with a set thats too big is that we can use the aforementioned principle to prove the existence of a whole slew of smaller sets, some of which will be paradoxical, like the aforementioned $\{x : x \notin x\}$. So, this is the usual approach to (hopefully) avoiding the paradoxes, and it requires that big collections, like the class of all topological spaces, cannot exist. For a similar reason, there is no "set of all things," nor a "set of all sets," nor anything like that.

Of course, there are other set theories where the "set of all sets" actually does exist. This is true in NFU, for example; and, if I'm not mistaken, it is also the case that the set of all topological spaces exists according to NFU. However, these sorts of set theories tend to have their own issues, which is why most people tend to prefer relatively "tame" ZFC (and its extensions) over the relatively "crazy" set theories like NFU.

In conclusion, its true that there is no set of all topological spaces, at least in "tame" theories like ZFC. However, the fundamental reason for this is issues of size, not issues of self-membership.

The statement:

A set cannot be a member of itself.

is a consequence of the so-called Axiom of Foundation or Axiom of Regularity. Other consequences of this axiom are, for example, that we can't have the following situations:

• Membership loops: $X \in Y \in X$
• Infinite membership chains: $X \ni Y \ni Z \ni \cdots$

However, it is nowadays known that a lot of standard mathematics can be done in the so-called $\sf ZF^-$ setting (the Zermelo-Fraenkel axioms without Foundation).

In the situation of a "set of all topological spaces", we run into a more critical problem than that of a set being an element of itself, namely Russell's Paradox. It shows its face when we want to select from our "set" $\mathscr T$ of topological spaces the collection:

$$\mathscr F = \{T \in \mathscr T \mid T \text{ is not a point of itself}\}$$

Then this $\mathscr F$ induces a subspace of $\mathscr T$ (in the discrete topology, say), and we are brought to Russell's paradox.

This paradox, however, is linked to the observation that "$\mathscr T$ is too large to be a set", as opposed to "$\mathscr T$ contains itself".

The axiomatic set theory $\sf ZF$ overcomes this by restricting the Axiom of Comprehension (also called "Axiom of Specification"), which specifies how we may form new sets, in a suitable way.

You might want to read into Russell's paradox. Basically, the problem is:

Let $S$ be the set of all sets which do not contain themselves.

This leads to a contradiction since $S$ cannot contain itself, nor can it not contain itself. Thus in general you can't simply say "let $S$ be the set of all sets with $Y$ property", or else you could run into a contradiction of this kind.

ZFC set theory avoids this problem by requiring that no set can be an element of itself, via the axiom of regularity. So the collection of all topological spaces cannot be a set, or else we could form a topological space on it, thus making it an element of itself.