Set builder notation: Colon or Vertical Line

There is no difference that I've ever heard of. I do strongly prefer "$\vert$" to "$\colon$", though, because I'm often interested in sets of maps, and e.g. $$\{f \mid f\colon \mathbb{R}\rightarrow\mathbb{C}\text{ with $f(6)=24$}\}$$ is easier to read than $$\{f: f\colon \mathbb{R}\rightarrow\mathbb{C}\text{ with $f(6)=24$}\}$$.

EDIT: Note that as Mike Pierce's answer shows, sometimes "$:$" is clearer. At the end of the day, use whichever notation is most clear for your context.

There is no difference. The bar is just often easier to read than the colon (like in the example in Noah Schweber's answer). However in analysis and probability, the bar is used in other notation. In analysis it is used for absolute value (or distance or norms) and in probability it is used in conditional statements (the probability of $A$ given $B$ is $\operatorname{P}(A \mid B)$). So looking at bar versus colon in sets with these notations $$ \{x \in X \mid ||x|-|y_0||<\varepsilon\} \quad\text{vs}\quad \{x \in X : ||x|-|y_0||<\varepsilon\} $$ $$ \{A \subset X \mid \operatorname{P}(B \mid A) > 0.42\} \quad\text{vs}\quad \{A \subset X : \operatorname{P}(B \mid A) > 0.42\} $$ it can be better to use the colon just to avoid overloading the bar.

It seems many good answers and examples have been provided, but I didn't see it mentioned above that regardless of which symbol you choose to use, in the set builder context both would be read aloud as "such that".

As for my example of preference, I am often working in number theory so I use the colon, ":", to avoid the bar being misinterpreted as "divides", or aesthetic reasons if I am also going to use the bar to mean divides in the specification of the set.