How to prove that the collection of rank-k matrices forms a closed subset of the space of matrices?
We can identify $M$ with $\mathbb{R}^{mn}$, by having the matrix $X=\pmatrix{a_{11} & \cdots & a_{1n}\\ \vdots & \ddots & \vdots \\ a_{m1} & \cdots & a_{mn}}$ correspond to the ordered tuple $(a_{11},a_{12},\ldots,a_{mn})$.
Let $r\leq\min\{m,n\}$, and let $M_r$ be the space of all $r\times r$ matrices. The map that sends an $m\times n$ matrix $X\in M$ to the $r\times r$ matrix formed from $X$ by removing all but a certain choice of $r$ rows (the $i_1,\ldots,i_r$'th rows, say) and a certain choice of $r$ columns (the $j_1,\ldots,j_r$'th colums, say), $$X=\pmatrix{a_{11} & \cdots & a_{1n}\\ \vdots & \ddots & \vdots \\ a_{m1} & \cdots & a_{mn}}\mapsto \pmatrix{a_{i_1j_1} & \cdots & a_{j_1j_r}\\ \vdots & \ddots & \vdots \\ a_{i_rj_1} & \cdots & a_{i_rj_r}}$$ corresponds to the map from $\mathbb{R}^{mn}$ to $\mathbb{R}^{r^2}$ that sends $$(a_{11},a_{12},\ldots,a_{mn})\mapsto (a_{i_1j_1},a_{i_1j_2},\ldots,a_{i_rj_r}),$$ which is continuous, for any choice of the $i$'s and $j$'s.
Furthermore, the determinant map from $M_r$ to $\mathbb{R}$ that sends $Y\in M_r$ to $\det(Y)$ is continuous, because it is a polynomial in the entries of $Y$, i.e. it corresponds to a map from $\mathbb{R}^{r^2}$ to $\mathbb{R}$ that is a polynomial in $r^2$ variables.
Suppose $k<\min\{m,n\}$. Then map $M$ to $\mathbb{R}^d$, where $d={\binom{m}{k+1}\cdot\binom{n}{k+1}}$, by sending a matrix to the ordered tuple of its $(k+1)\times(k+1)$ minors. For example, if $m=n=3$ and $k=1$, then we send $$\pmatrix{1 & 2 & 3\\ 4 & 5 & 6\\ 7 & 8 & 9}\mapsto\left( |\begin{smallmatrix} 1 & 2\\ 4 & 5\end{smallmatrix}|, |\begin{smallmatrix} 1 & 3\\ 4 & 6\end{smallmatrix}|, |\begin{smallmatrix} 2 & 3\\ 5 & 6\end{smallmatrix}|, |\begin{smallmatrix} 1 & 2\\ 7 & 8\end{smallmatrix}|, |\begin{smallmatrix} 1 & 3\\ 7 & 9\end{smallmatrix}|, |\begin{smallmatrix} 2 & 3\\ 8 & 9\end{smallmatrix}|, |\begin{smallmatrix} 4 & 5\\ 7 & 8\end{smallmatrix}|, |\begin{smallmatrix} 4 & 6\\ 7 & 9\end{smallmatrix}|, |\begin{smallmatrix} 5 & 6\\ 8 & 9\end{smallmatrix}| \right).$$
Because all $d$ of the projection maps from $M$ to $M_r$ are continuous, and because the determinant map from $M_r$ to $\mathbb{R}$ is continuous, this map is continuous.
A matrix is of rank $\leq k$, i.e. $< k+1$, if and only if all of its $(k+1)\times(k+1)$ minors are zero. Thus, $C$ is the preimage of $0\in\mathbb{R}^d$ under a continuous map, and $\{0\}$ is a closed set in $\mathbb{R}^d$ because $\mathbb{R}^d$ is Hausdorff, so therefore $C$ is a closed set in $M$.
If $k=\min\{m,n\}$, then $C=M$, and is a closed set.
Zev Chonoles's answer is totally fine, but since we don't need to care about redundancy, we may simply look at $$f:\mathbb R^{m\times n}\to\mathbb R^{m!n!}\;,\;\;\;A\mapsto\left(\det\left(A_{\pi_1(i)\pi_2(j)}\right)_{1\:\le\:i,\:j\:\le k}\right)_{\substack{\pi_1\:\in\:S_m\\\pi_2\:\in\:S_n}}$$ instead, where $S_n$ denotes the symmetric group. Then, $f$ maps $A$ to the determinants of the submatrix given by the first $k$ columns and rows for every possible permutation of the columns and rows of $A$. And clearly $C=f^{-1}(\{0\})$.