Intuition for Kuratowski-Mrówka characterization of compactness
What does the closedness of the projection $\pi : X \times Y \to Y$ really mean?
That the compactness of $X$ implies that every projection $\pi : X \times Y \to Y$ is closed really stems from the Tube Lemma, and a characterisation of closed (continous) maps.
Tube Lemma. (See Lemma 26.8 from Munkres's Topology.) If $X$ is compact, $Y$ is any space, $y_0 \in Y$, and $W \subseteq X \times Y$ is open such that $X \times \{ y_0 \} \subseteq W$, then there is an open neighbourhood $V$ of $y_0$ such that $X \times V \subseteq W$.
A continuous function $f : X \to Y$ is closed iff for each $y \in Y$ and each open $U \subseteq X$ with $f^{-1} [\{ y \}] \subseteq U$ there is an open neighbourhood $V$ of $y$ such that $f^{-1} [V] \subseteq U$.
Restating the latter in the special case of the projection $\pi : X \times Y \to Y$ we get
- $\pi : X \times Y \to Y$ is closed iff for each $y \in Y$ and each open $W \subseteq X \times Y$ with $X \times \{ y \} = \pi^{-1} [\{y\}] \subseteq W$ there is an open neighbourhood $V$ of $y_0$ such that $X \times V = \pi^{-1} [ V ] \subseteq W$.
So the statement that the projection $\pi : X \times Y \to Y$ is closed for every $Y$ is really just a restatement of the Tube Lemma for $X$.
"Intuition" for the sufficiency of the statement
The Tube Lemma is probably easier to gain an intuition for. It basically says that if $W \subseteq X \times Y$ is open and contains a segment $X \times \{ y_0 \}$ (which is homeomorphic to $X$), then it can't be too "erratic" or become arbitrarily "thin" around that segment.
If $X$ is compact, then since $\langle x,y_0 \rangle \in W$ there are open $U_x \subseteq X$ and $V_x \subseteq Y$ such that $\langle x,y_0 \rangle \in U_x \times V_x \subseteq W$. If $X$ is compact, we only need finitely many of the $U_x$s to cover $X$, and the intersection of the corresponding $V_x$s will yield $V$.
If $X$ is not compact, you can imagine that the $U_x$s form an open cover with no finite subcover, and so when you intersect the corresponding $V_x$s for a family of the $U_x$s which do cover $X$, you might not get an open set as desired.
The trouble then is picking out an appropriate space to witness the latter for each non-compact space.
Encoding information about an open cover in an auxiliary space
That the closedness of each projection $\pi : X \times Y \to Y$ implies compactness rests on constructing an auxiliary topological space $Y$ for a given family $\mathcal{U}$ of open subsets of $X$ in such a way that properties of space $Y$ and the projection $\pi : X \times Y \to Y$ encode facts about the family $\mathcal{U}$. In particular, $\pi$ being closed will imply that either $\mathcal{U}$ does not cover $X$, or that a finite subfamily covers $X$.
The actual construction of $Y$ is a bit synthetic, and may not yield to intuition. But we can try to pick apart some ideas.
Consider the general case of a topological space $X$, and a family $\mathcal{U}$ of open subsets of $X$. We consider the set $Y = X \cup \{ \mathord{*} \}$ where $\mathord{*} \notin X$ with the topology generated by the basis consisting of
- $\{ x \}$ for each $x \in X$; and
- all sets of the form $\{ \mathord{*} \} \cup ( X \setminus ( U_1 \cup \cdots \cup U_n ) )$ where $U_1 , \ldots , U_n \in \mathcal{U}$.
Central Fact. For $A \subseteq Y$, $\mathord{*} \in \overline{A}$ iff $A$ cannot be covered by finitely many sets from $\mathcal{U}$.
If the projection $\pi : X \times Y \to Y$ is closed, then in particular the implication $$\mathord{*} \in \overline{ \pi[\Delta] } \Rightarrow \mathord{*} \in \pi[ \overline{\Delta} ]$$ holds, where $\Delta = \{ \langle x,x \rangle : x \in X \} \subseteq X \times Y$.
As $\pi[\Delta] = X$, we have that $\mathord{*} \in \overline{\pi[\Delta]}$ iff $X$ cannot be covered by finitely many sets from $\mathcal{U}$.
Now $\mathord{*} \in \pi [ \overline{\Delta} ]$ iff there is an $x \in X$ such that $\langle x , \mathord{*} \rangle \in \overline{\Delta}$.
For $x \in X$, if $x \in \bigcup \mathcal{U}$, then take $U \in \mathcal{U}$ containing $x$. Clearly $U$ is an open neighbourhood of $x$ in $X$, and $\{ \mathord{*} \} \cup ( X \setminus U )$ is an open neighbourhood of $\mathord{*}$ in $Y$, however $( U \times ( \{ \mathord{*} \} \cup ( X \setminus U ) ) ) \cap \Delta = \varnothing$, so $\langle x , \mathord{*} \rangle \notin \overline{\Delta}$. On the other hand, if $x \notin \bigcup \mathcal{U}$, then as every open neighbourhood of $\mathord{*}$ contains $x$ it follows that $\langle x , \mathord{*} \rangle \in \overline{\Delta}$.
So for $x \in X$ we have that $\langle x,\mathord{*} \rangle \in \overline{\Delta}$ iff $x \notin \bigcup \mathcal{U}$, and so $\mathord{*} \in \pi[\overline{\Delta}]$ iff $\bigcup \mathcal{U} \neq X$.
Putting this together, in order for $\pi$ to be closed it must be the case that either $X$ can be covered by finitely many sets from $\mathcal{U}$, or that $\mathcal{U}$ does not cover $X$.
We can use the same space $Y$ to investigate how the Tube Lemma for $X$ implies the compactness of $X$. Note that $W = ( X \times Y ) \setminus \overline{ \Delta }$ is an open subset of $X \times Y$. For the Tube Lemma for $X$ to hold, it must be that the implication
$X \times \{ \mathord{*} \} \subseteq W$ ⇒ $X \times V \subseteq W$ for some open neighbourhood $V$ of $\mathord{*}$
holds.
- One can show that $X \times \{ \mathord{*} \} \subseteq W$ iff $\bigcup \mathcal{U} = X$. (In particular, as above, $\langle x, \mathord{*} \rangle \in \overline{\Delta}$ iff $x \notin \bigcup \mathcal{U}$.)
- If $V$ is an open neighbourhood of $\mathord{*}$ such that $X \times V \subseteq W$, then without loss of generality there are $U_1 , \ldots , U_n \in \mathcal{U}$ such that $V = \{ \mathord{*} \} \cup ( X \setminus ( U_1 \cup \cdots \cup U_n ) )$. It then follows that $U_1 \cup \cdots \cup U_n = X$.
Some concrete examples might be helpful. Let's choose $X = Y = \mathbb{R}$. Now the graph of $\arctan: \mathbb{R} \to (-\pi/2, \pi/2)$ is a closed subset of $X \times Y$. However, the projection $X \times Y \to Y$ is not a closed map since the projection of the graph is $(-\pi/2, \pi/2)$. This demonstrates that $\mathbb{R}$ is not compact.
So if $X$ is not compact, the "shadow" of a closed set in $X \times Y$ might not be closed.
There are many more examples in $\mathbb{R}^2$ that help. If we have some horizontal line, and a closed subset of $\mathbb{R}^2$ that asymptotically approaches this horizontal line, then the projection to the $y$-axis does not contain the point where this line intersects the $y$-axis, but contains everything below (or above) it. Thus the projection is not closed. But if $X$ is some compact subset of the $x$-axis, then a closed subset of $X \times \mathbb{R}$ can't asymptotically approach a vertical line.