Examples of Galois connections?
There is a nice section on Galois connections in George Bergman's An Invitation to General Algebra and Universal Constructions (available from his website). If you want to learn more about Galois connections, I highly recommend reading the entire chapter (except perhaps section 5.4, which is a complete digression).
The general setting is:
Let $S$ and $T$ be sets, and let $R\subseteq S\times T$ be a relation from $S$ to $T$. For any $A\subseteq S$ and $B\subseteq T$, let $$\begin{align*} A^* &= \{t\in T\mid \forall a\in A (aRt)\}\subseteq T\\ B^* &= \{s\in S\mid \forall b\in B (sRb)\}\subseteq S. \end{align*}$$ This gives us two functions, one from $\mathcal{P}(S)$ to $\mathcal{P}(T)$, and one from $\mathcal{P}(T)$ to $\mathcal{P}(S)$. The operations are:
Inclusion reversing: if $A\subseteq A'\subseteq S$ and $B\subseteq B'\subseteq T$, then $(A')^*\subseteq A^*$ and $(B')^*\subseteq B^*$.
Increasing: $A\subseteq A^{**}$ for all $A\subseteq S$ and $B\subseteq B^{**}$ for all $B\subseteq T$.
For all $A\subseteq S$, $A^{***}=A^*$; for all $B\subseteq T$, $B^{***}=B^*$.
In particular, the maps $A\mapsto A^{**}$ and $B\mapsto B^{**}$ give closure operators on $\mathcal{P}(S)$ and $\mathcal{P}(T)$. The closed elements of $\mathcal{P}(S)$ are precisely the sets of the form $B^*$, the closed elements of $\mathcal{P}(T)$ are precisely the sets of the form $A^*$, and the $*$ operation restricted to closed sets gives an anti-isomorphism (order-reversing bijection whose inverse is also order-reversing) between the complete lattice of ${}^{**}$-closed subsets of $S$ and of $T$.
Bergman notes:
A Galois connection between two sets $S$ and $T$ becomes particularly valuable when the ${}^{**}$-closed subsets have characterizations of independent interest.
Here are the examples he gives:
The "classical example". Let $S$ be the underlying set of a field $F$, $T$ the underlying set of a finite group $G$ of automorphisms of $F$. For $a\in F$ and $g\in G$, let $aRg$ mean "$g$ fixes $a$" (i.e., $g(a)=a$). The Fundamental Theorem of Galois Theory says that the closed subsets of $F$ are precisely the subfields of $F$ that contain the set $G^*$, and that the closed subsets of $G$ are precisely all subgroups of $G$.
Let $S$ be a vector space over a field $K$, and $T$ the dual space $\mathrm{Hom}_K(S,K)$. Let $xRf$ mean "$f(x)=0$". The closed subsets of $S$ are precisely the vector subspaces, the closed subsets of $T$ are precisely the vector subspaces that are closed in a certain topology (all subspaces when the dimension is finite, but when the dimension is infinite you get interesting stuff).
Let $S=\mathbb{C}^n$ (complex affine $n$-space), and $T=\mathbb{Q}[x_0,\ldots,x_{n-1}]$, the polynomial ring in $n$ indeterminates with rational coefficients. Let $(a_0,\ldots,a_{n-1})Rf$ mean $f(a_0,\ldots,a_{n-1})=0$. This is the starting point of classical algebraic geometry: the closed subsets of $\mathbb{C}^n$ are the solution sets of polynomial equations, and the Nullstellensatz characterizes the closed subsets of $T$ as the radical ideals.
Let $S$ be a finite dimensional real vector space, $T$ the set of pairs $(f,a)$ where $f$ is a linear functional on $S$ and $a\in\mathbb{R}$. Let $xR(f,a)$ mean $f(x)\leq a$. The closed subsets of $S$ are the convex sets.
Let $S$ be a finite dimensional real vector space, $T$ the set of linear functionals on $S$. Let $xRf$ mean $f(x)\leq 1$. The closed subsets on each side are the convex subsets that contain the zero vector.
Let $A$ be an abelian group (or a module over a commutative ring) and $S=T=\mathrm{End}(A)$ the ring of endomorphisms. Let $sRt$ stand for $st=ts$. Given a subring $X$ of $S$, $X^*$ is the commutant of $X$, an important subring studied by ring theorists.
Let $S$ be a set of mathematical objects, $T$ a set of propositions about objects of this sort, and $sRt$ stand for "object $s$ satisfies proposition $t$" (for logicians, $s\models t$). The closed subsets of $S$ are the axiomatic classes, the closed subsets of $T$ are the theories.
Added. It seems to me that Baez is staking a middle ground between the full generality of Galois connections and the special case of Galois Theory of fields, by considering only the situation in which $T$ is a set, $S$ is a subgroup of the group of bijections $T\to T$, and the relation is $fRt$ if and only if $f(t)=t$. In that case, you can always let $k=S^*$, and you do indeed get a correspondence between the closed subsets of $T$ and the closed subgroups of $S$ (though not every subgroup need be closed; this occurs for instance in the Galois group of an infinite field extension, where only subgroups that are closed in the profinite topology correspond to subfield of the extensions).
If you take the lattice of ideals of a ring $R$ and the lattice of ideals of $R/\mathfrak{a}$ for some ideal $\mathfrak{a}$ then the maps $\mathfrak{b}\mapsto\pi(\mathfrak{b})$ and it's inverse form a monotone Galois connection between the ideals of $R$ containing $\mathfrak{a}$ and the ideals of $R/\mathfrak{a}$.
I'm sure you can make a similar guess for the case of subgroups of groups, submodules of modules, etc.
More generally, a Galois connection between two posets $I$ and $J$ is nothing more than an adjoint pair of functors between their corresponding categories. This allows you to create a whole host of examples.
Let $(\Omega,\Sigma, \mathbb P)$ be a probability space and let $X : \Omega \to \mathbb R$ be a random variable. Consider the cumulative distribution function: $$F_X : [-\infty\,.. \infty] \to [0\,.. 1], x\mapsto \mathbb P (X\leq x)$$
This is a monotone map and it always has left-adjoint $Q_X$ called the "quantile function" given by:
$$Q_X : [0\,.. 1] \to [-\infty\,.. \infty] , p\mapsto \inf\{x\in \mathbb R : p< F(x)\}$$
For example the median of $X$ would be $Q_X(\frac{1}{2})$.
Since adjoints are unique (up to natural isomorphism, but here we have posets) the quantile function determines the distribution of $X$ much like the CDF, PDF, PMF or characteristic function.
The quantile function is also known as "inverse distribution function", "generalized inverse function" and so on and so forth. As far as I can guess it is simply not well-known that this is simply an instance of a Galois connection.