Continuous relations?

Here is an expansion of my comment into an answer which I think is very compelling as the "correct" definition for compact Hausdorff spaces, though I agree with others who have said that for general spaces there may be several competing definitions with different merits. My argument for this being the right definition is that it is natural in two different ways: it arises naturally by taking the definition of "homomorphism" and modifying it in an obvious way to apply to relations, and it also coincides with the categorical definition of relations as subobjects of the product $X\times Y$. Furthermore, the coincidence of these two definitions occurs very generally (in particular, in any category monadic over sets).

Let me start by considering this question in different (concrete) categories. For instance, what might it mean for a relation $R\subseteq G\times H$ to be "homomorphic"? If you think of a relation as a multivalued function, the following definition seems pretty reasonable: for any $g,g'\in G$, if $h$ is a value of $R(g)$ and $h'$ is a value of $R(g')$, then $hh'$ should be a value of $R(gg')$. We should also demand that $1$ is a value of $R(1)$ and that if $h$ is a value of $R(g)$, then $h^{-1}$ is a value of $R(g^{-1})$ (demanding these is redundant for functions but not for relations). It is then easy to check that this is actually equivalent to $R\subseteq G\times H$ being a subgroup of the product group. This easily generalizes to any other sort of algebraic object: there is an analogous definition of "homomorphic relation", and it is equivalent to being a subobject of the product.

What, then, is the analogue for topological spaces? Well, if you want to think of a space as a set with some sort of "operations" on it, those operations should be taking limits. Because limits neither always exist nor are unique in general, there are a few different ways you might define what it means for a relation to preserve limits. The following is the one I have found to be most natural:

(1)$\,$a relation $R\subseteq X\times Y$ is continuous if whenever $x$ is an accumulation point of a net $(x_a)$ in $X$ and $y_\alpha$ is a value of $R(x_\alpha)$, then there is some accumulation point of $(y_\alpha)$ that is a value of $R(x)$.

Equivalently, we could restrict to universal nets and replace "accumulation point" with "limit" everywhere (however, unlike for functions, it is not equivalent to consider arbitrary nets and replace "accumulation point" with "limit", because there might be values of $R(x)$ that are limits of every universal subnet but no single value that is simultaneously a limit of all of them).

This definition has advantages and disadvantages. A function is continuous as a relation iff it is continuous in the usual sense and a composition of continuous relations is continuous. A partial function that is continuous on its domain is continuous as a relation iff its domain is closed. However, this definition is not symmetric in $X$ and $Y$ (as Joonas Ilmavirta observed, this is a necessary consequence of agreeing with the usual definition on functions). It also does not coincide with subobjects of $X\times Y$ in the category of topological spaces (which include not only all subspaces of $X\times Y$ but also all subsets equipped with any finer topology).

However, if we restrict to compact Hausdorff spaces, the disadvantages disappear. Limits of universal nets or ultrafilters are well-defined single-valued operations on compact Hausdorff space, so there is a clear choice for what it means for a relation to be "homomorphic with respect to limits". A relation between compact Hausdorff spaces is continuous iff it is closed as a subset of $X\times Y$, and thus continuity is symmetric in $X$ and $Y$. In addition, these continuous relations are also exactly those subsets of $X\times Y$ that are themselves compact Hausdorff spaces, just as in the case of homomorphic relations between algebraic structures.

As a final note, there is a simultaneous generalization of the algebraic case and compact Hausdorff spaces, which is algebras over a monad (compact Hausdorff spaces are the same as algebras over the monad that takes a set to the set of ultrafilters on it, with the structure map of an algebra telling you how to take limits of ultrafilters). Let $T:\mathrm{Set}\to\mathrm{Set}$ be a monad and let $A$ and $B$ be sets. Given a relation $R\subseteq A\times B$, we can consider the two projections $A\leftarrow R\to B$ and apply $T$ to get a diagram $TA\leftarrow TR\to TB$. Let $\tilde{T}R$ be the image of $TR$ in the product $TA\times TB$. In this way, $T$ naturally extends to a functor $\tilde{T}:\mathrm{Rel}\to\mathrm{Rel}$.

We can now define a "homomorphic relation" between $T$-algebras. Let $A$ and $B$ be $T$-algebras with structure maps $\mu_A:TA\to A$ and $\mu_B:TB\to B$. We say a relation $R\subseteq A\times B$ is homomorphic if for any $x\in TA$, if $y$ is a value of $\tilde{T}R(x)$, then $\mu_B(y)$ is a value of $R(\mu_A(x))$. But this is just saying that $\mu_A\times \mu_B:TA\times TB\to A\times B$ restricts to a map $\tilde{T}R\to R$, and this restriction will then make $R$ itself a $T$-algebra via the composition $TR\to \tilde{T}R\to R$ and a subalgebra of $A\times B$. Conversely, if $R$ is a subalgebra of $A\times B$, then the structure map $TR\to R$ must factor through $\tilde{T}R$ as a restriction of $\mu_A\times \mu_B$. Thus homomorphic relations between algebras over a monad always coincide with subalgebras of the product.

I have a couple of remarks regarding continuity and some natural constructions of relations:

• Unlike functions, relations have no preferred direction. So if $R\subset X\times Y$ is a relation, its inverse relation $R^{-1}\subset Y\times X$ is an equally valid relation. Now if we want the concept of a continuous relation to respect this symmetry ($R$ is continuous iff $R^{-1}$ is), we have a significant restriction. The obvious attempt to define a continuous relation so that the preimage of any open set needs to be (relatively) open leads to a nonsymmetric concept; it is easier to generalise open continuous functions symmetrically. In fact, a generalization of continuous functions cannot be symmetric (without additional structural assumptions on the spaces) since there are continuous bijections without continuous inverse.

• Let $R\subset X\times Y$ be a relation. The preimage $R^{-1}Y\subset X$ need not be all of $X$ (unlike for functions). If we define $R$ to be continuous when the preimage of every open set is open, a partial function obtained by dropping part of a continuous function need not be continuous. This seems weird (but may be inevitable). One could also demand that the preimage of any set relatively open in $RX$ (or just open in $Y$ if it seems better) is relatively open in $R^{-1}Y$.

• If $R\subset X\times Y$ and $S\subset Y\times Z$ are relations, their composition $S\circ R=\{(x,z);\exists y\in Y:xRySz\}\subset X\times Z$ is a relation. If $R$ and $S$ are continuous, it would seem natural to require that $S\circ R$ be continuous as well. This poses restrictions on the definitions presented in the previous remark; it could happen, for example, that $S^{-1}Z\cap RX\subset Y$ is empty or somehow bad (neither open nor closed). If we define a continuous relation so that the preimage of an open set must be open, composition preserves continuity, but passing to partial functions does not. The composition of two (usual/partial/multivalued) functions is again a (usual/partial/multivalued) function, so I think respecting composition is a good idea.

It seems that we can't keep all the good properties of continuous functions and ordinary relations in a theory of continuous relations. Therefore different applications will probably call for different definitions. (This vacuously true if there is at most one application.)

Since the question is open-ended and basically just seems to be a request for cool ideas, is it okay if I answer a slightly different question? Namely: what is the "right" notion of a measurable relation?

The obvious answer --- take $X$ and $Y$ to be measure spaces and $R$ to be a measurable subset of $X \times Y$ --- is badly behaved. If $X$ is nonatomic then the reflexivity condition for relations on $X$ becomes vacuous, and making sense of transitivity is also problematic.

But there is a good answer! Work with positive measure subsets modulo null sets and assume $X$ and $Y$ are $\sigma$-finite, so we can take joins of arbitrary families of positive measure subsets. Then we characterize measurable relations by saying which pairs of positive measure subsets belong to the relation. The condition is: a measurable relation is a family $R$ of ordered pairs of positive measure subsets of $X$ and $Y$ such that $$\big(\bigvee A_\alpha, \bigvee B_\alpha\big) \in R\qquad \Leftrightarrow\qquad \mbox{some }(A_\alpha, B_\alpha) \in R,$$ for any families $\{A_\alpha\}$ and $\{B_\alpha\}$ of positive measure subsets of $X$ and $Y$, respectively. The intuition is that a pair $(A,B)$ belongs to the relation if and only if some point of $A$ is related to some point of $B$.

There is a well-developed theory of measurable relations in this sense. They can be composed, for example. The diagonal relation $\Delta$ is defined by setting $(A,B) \in \Delta$ iff $A \cap B$ is nonnull, and a relation is reflexive if it contains $\Delta$, etc. The details are given in Section 1 of this paper of mine.

Interestingly, as far as I know, there is no good definition of the complement of a measurable relation in this sense.