No continuous function switches $\mathbb{Q}$ and the irrationals

Here's a way to use connectedness, really amounting to using the intermediate value theorem.

If $f(\mathbb{Q})\subseteq \mathbb R\setminus\mathbb Q$ and $f(\mathbb R\setminus \mathbb Q)\subseteq\mathbb Q$, then $f(0)\neq f(\sqrt 2)$. Because intervals are connected in $\mathbb R$ and $f$ is continuous, $f[0,\sqrt 2]$ is connected. Because connected subsets of $\mathbb R$ are intervals, $f[0,\sqrt 2]$ contains the interval $\left[\min\{f(0),f(\sqrt 2)\},\max\{f(0),f(\sqrt 2)\}\right]$. The set of irrational numbers in this interval is uncountable, yet contained in the countable set $f(\mathbb Q)$, a contradiction.

A slightly briefer outline: The hypothesis implies that $f$ is nonconstant with range contained in the countable set $\mathbb Q\cup f(\mathbb Q)$, whereas the intermediate value theorem and uncountability of $\mathbb R$ imply that a nonconstant continuous function $f:\mathbb R\to\mathbb R$ has uncountable range.

Suppose there is such a mapping $f$. Consider $g:[0,1]\to \mathbb{R}$ defined by $$g(x)=f(x)-x.$$ Suppose that $g(x)\in \mathbb{Q}$ for some $x\in [0,1]$. Then:

• if $x\in J$, then $g(x)-f(x)\in \mathbb{Q}$, i.e. $x\in \mathbb{Q}$.
• if $x\in \mathbb{Q}$, then $g(x)+x\in \mathbb{Q}$, i.e. $f(x)\in \mathbb{Q}$, but $f(x)\in J$.

Both produce contradictions. Thus $g([0,1])\subseteq J$. Since $f$ is continuous, $g$ is continuous, and then $g([0,1])=[\min g,\max g]$. If $g$ is not constant then there exists $r$ a rational in $[\min g,\max g]$. By the intermediate value theorem, there exists $z\in[0,1]$ such that $g(z)=r$, but this is impossible because $g([0,1])\subseteq J$. Therefore, $g$ must be constant and then $$f(x)=c+x$$ with $c\in J$. Particularly, $f(c)=2c\not\in \mathbb Q$ which, as Jonas pointed, is contradictory to the hypothesis. Therefore such an $f$ can not exist.

Another simple proof:

Because $f$ is continuous and by connectedness, $f([0,1])=[a,b]$ for some $a<b$. Now define $$g : x \mapsto \frac{1}{p} \left(f(x)-q \right), \ \text{where} \ p,q \in \mathbb{Q}.$$

In particular, $g(x)$ is rational iff $f(x)$ is rational, i.e. $g$ has the same property that $f$.

Notice that $g([0,1])= \left[ \frac{a-q}{p}, \frac{b-q}{p} \right]$. Therefore, if $b-1 \leq q \leq a$ and $p \geq b-q$ then $g : [0,1] \to [0,1]$ and classically $g$ has a fixed point $x_0 \in [0,1]$.

Finally we deduce that $x_0 \in \mathbb{Q}$ iff $x_0 = g(x_0) \notin \mathbb{Q}$, a contradiction.