Why does \big\mid not work?

• The commands \bigl\vert, \bigm\vert, and \bigr\vert are semantically symmetric. Incidentally, the triplets of commands \bigl\lvert \bigl\vert \big\lvert and \bigr\rvert \bigr\vert \big\rvert, respectively, produce the same output.

• It's the command \mid that's a bit of an outlier, semantically speaking. As @egreg has noted in a comment, \mid is constructed as a relation symbol and is not set up to take a size-modifying prefix.

\documentclass{article}
\usepackage{amsmath}
\begin{document}

\dots\ $\bigl\lvert \ldots \bigm| \ldots \bigr\rvert$ \dots\ works fine.

\dots\ $\bigl\vert \ldots \bigm\vert \ldots \bigr\vert$ \dots\ works just the same.

\dots\ $\bigl\vert \ldots \big| \ldots \bigr\vert$ \dots\ works too, but it isn't the same.

\end{document}

Using \big is generally wrong, because it produces an ordinary atom. So one has better using \bigl for opening fences, \bigr for closing and \bigm for relations.

The definition of \bigX ultimately does \big anyway, but first adding the correct type. And \big<token> simply does

\left<token>

so we must ensure that the argument to \bigX is a delimiter, which \mid isn't.

One could, in principle, lift off this limitation for relations that we know are built upon a delimiter:

\documentclass{article}
\usepackage{amsmath}

\makeatletter
\let\amsmath@bigm\bigm

\renewcommand{\bigm}[1]{%
  \ifcsname fenced@\string#1\endcsname
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
  {\expandafter\amsmath@bigm\csname fenced@\string#1\endcsname}%
  {\amsmath@bigm#1}%
}

\newcommand{\DeclareFence}[2]{\@namedef{fenced@\string#1}{#2}}
\makeatother

\DeclareFence{\mid}{|}

\begin{document}

$\bigl\{\, x\in X \bigm\mid x\notin X \,\bigr\}$

\smallskip

$\bigl\{\, x\in X \bigm| x\notin X \,\bigr\}$

\end{document}

This is just a proof of concept, so I didn't attempt a generalization to \bigm siblings \Bigm, \biggm and \Biggm.