Why $\{x: x = x\}$ is not a set in Naive Set Theory? (Halmos, Sec. 4)

I agree with you that Halmos's sentence (page 11) :

In case $S(x)$ is $(x∈′x)$, or in case $S(x)$ is $(x=x)$, the specified $x$'s do not constitute a set.

is not crystal clear.

Halmos has introduced the Specification axiom (page 6):

for every $A$ and condition$S(x)$ the set $B$ whose elements... witten $B= \{ x \in A : S(x) \}$ exists.

Then (same page) he proves, using the construction of Russell's paradox, that there is no "universal" set; i.e. he proves that:

for every set $A$, there is a set $B$ such that $B \in' A$,

specifically, this set is $B = \{ x \in A : x \in ' x \}$.

Comment: here the definition of $B$ is correct, according to Specification. What we have to note is that, up to now (page 6), we do not know if there are sets at all.

Specification is a "conditional" set existence axiom: if we have a set $A$, the axiom licences us to build new sets (subsets of $A$) for every "specifiable" condition $S(x)$.

Then (page 8) we start populating the "universe" of sets, with the "temporary" axiom:

there exists a set.

In other axiomatization of the theory, the first "existential axiom" is usually the Empty set axiom.

In Halmos approach, the existence of the empty set is proved: let $A_0$ the set whose existence has been asserted by the above axiom; then, by Specification, we have that:

$\{ x \in A_0 : x \ne x \}$

exists. Clearly, this set is empty, and by Extensionality we have that it is unique (i.e. two empty sets must coincide) and thus we call it the emptyset : $\emptyset$.

Third axiom (page 9) : Pairing. Again, it is "conditional" : for any two sets $a, b$ there exists a set that they both belong to :

for every $a,b$, the set $A = \{ x = a \text { or } x = b \}$ exists;

or: $x \in A \text { iff } x = a \text { or } x = b$. Call it $\{ a, b \}$.

Comment : again, up to now we know only of $\emptyset$. Thus, applying Pair to it, what we get is nothing more than $\{ \emptyset \}$, and so on.

In general (page 10), Pair applied to $a$ only (instead of : $a,b$) gives us $\{ a \}$ for every set $a$.

And now we have the comments of page 10 (last half) and page 11: .

We know that $\{ x : S(x) \}$ is not a correct way to create sets.

In spite of this, Halmos consider some "good" cases :

(i) let $S(x)$ the formula $x \in A$.

In this case, the "bad" $\{ x : S(x) \}$ works, because it amounts to :

$\{ x : x \in A \} = \{ x \in A : x \in A \}=A$

and this is a legitimate instance of Specification.

Comment : of course, it is only a "typographical" usage because we cannot use it to prove nothing new. The set $A$ is "already there".

(ii) The same with $x \ne x$ as $S(x)$.

We have that:

for every $A$ : $\{ x \in A : x \ne x \} = \emptyset$

exists, by Specification. In this way, we have "specified" the empty subset of every set $A$; but by extensionality they all coincide, and thus we do not care about $A$.

Comment : maybe the "standard" approach of postulating directly the existence of $\emptyset$ is better...

(iii) Also the "wrong" $\{ x : x = a \}$ does not harm.

It is only a shorthand for the set $\{ x : x = a \text { or } x = a \}$ licensed by Pairing.

And here we stop...

Two further cases of $S(x)$ :

$(x \in ' x)$ and $(x=x)$

cannot be used, because they lead to problems (discussed above).

I hope it may help.

If $V=\{x:x=x\}$ is a set, then by specification, $\{x:x\notin x\}$ is also a set, since every $x=x$, so $\{x:x\notin x\}=\{x\in V:x\notin x\}$. Of course, this is impossible.

It should be noted that there are set theories with a universal set, but they resolve the problem of Russell's paradox in a different way. One such set theory is Quine's New Foundations.