Compact open topology

Given two spaces $X$ and $Y$, how to define the mapping space betweeen them, i.e. what topology should we put on the set of maps between them?

If $X$ is compact and $Y$ a metric space, this is quite easy as one can put a metric on $Map(X,Y)$: For $f,g\in Map(X,Y)$ define their distance just to be the maximum of the distances between $f(x)$ and $g(x)$ as $x$ ranging over the points in $X$.

If $Y$ is no longer metric, we have to find a replacement what it means for two maps to be close. Say, we have again two maps $f,g\in Map(X,Y)$. Let $K\subset X$ be compact and $U\subset Y$ be open such that $f(K)\subset U$. Assume now that $Y$ is Hausdorff (else, this construction might behave badly anyhow). Then $f(K)\subset Y$ is closed, so you would expect that if you move $f(K)$ a little bit, then it stays inside of $U$. So if $g$ is close to $f$, then $g(K)$ should be still inside of $U$. Thus, it is sensible to define an open neighborhood of $f$ to be all maps $g$ such that $g(K) \subset U$. And actually, this agrees with the metric definition when $X$ is compact and $Y$ metric! Furthermore, as long as you have "enough" compact subsets in $X$ (i.e. $X$ is compactly generated) and everything is Hausdorff, then this topology has all the pleasant properties one likes.

I have a weakness for pictures. So here is one

^(source)

The above shows a function $f: \mathbb R \to \mathbb R$, a compact set $C$, and an open set $U$. The condition $f(C) \subseteq U$ is that the graph of $f$ passes through the shaded part shown in the picture.

The compact open topology is essential for getting compact sets in your function space–especially a version of the Arzelà-Ascoli theorem holds for spaces of continuous functions to uniform spaces equipped with the compact open topology ($C(X,Y)$ with the compact open topology where $X$ is an appropriate topological space and $Y$ is an appropriate uniform space).

In your case (Pontryagin duality) the uniform space $Y$ is just the circle group $\mathbb{T}$ and $X$ is your group $G$. Consider a compact neighbourhood $U$ of the neutral element $e\in G$, an open neighbourhood $I:=\exp((-2\pi i\epsilon, 2\pi i\epsilon))\subset\mathbb{T}$ of 1 where $0<\epsilon<1$. Now there is a maximal set of irreducible representations $S\subset \hat{G}\subset C(G,\mathbb{T})$ such that $\forall \pi\in S\ \pi(U)\subset I$. The definition of the compact open topology guarantees that $S$ is open (it is just the definition). But $S$ is also equicontinuous: For $n\in\mathbb{N}$ define $U_0:=U$,$U_{n+1}:=\{x\in U_n\mid x+x\in U_n\}$. Clearly for $\pi\in S$ we have $\pi(U_n)\subset \exp((-2\pi i \epsilon/2^n,2\pi i \epsilon/2^n))$. It suffices to prove equicontinuity at the point $e$ (shifting does not change the situation). Now we can apply Arzelà-Ascoli: $S$ is in fact precompact and we get a compact neighbourhood $\bar{S}$ of the neutral element of $\hat{G}$ thus $\hat{G}$ is locally compact.

There is also a categorical motivation for this topology: The category of topological spaces is not cartesian closed, thus in this category (and many usual related topologies) the adjunction mentioned by David White in the comment above (currying) does not exist—it is impossible to choose the right topology. However, in the category of compactly generated spaces it works: It is cartesian closed and the topology for function spaces is the compact open topology. Again, the compact open topology guarantees that there are enough compact sets such that the topology is actually compactly generated (I guess that this is also a consequence of a version of Arzelà-Ascoli using even continuity, which does not need a uniformity, instead of equicontinuity, but I am not sure).