Why can't there be a monotone function with domain $\mathbb{R}$ and range $\mathbb{R} \setminus \mathbb{Q}$?

The basic reason that there can be no monotone mapping of $\mathbb{R}$ onto $\mathbb{R}\setminus\mathbb{Q}$ is the completeness of the linear order on $\mathbb{R}$.

Suppose that $f:\mathbb{R}\to\mathbb{R}\setminus\mathbb{Q}$ is an increasing surjection. Let $L=\{x\in\mathbb{R}:f(x)<0\}$; clearly $L$ is bounded above (e.g., by the real number $y$ such that $f(y)=\sqrt2$), so $L$ has a least upper bound $u$. Now what can $f(u)$ be?

• If $u\notin L$, then $f(u)>0$, and $f(x)\ge f(u) > 0$ for every $x\ge u$, so $f[\mathbb{R}]\cap(0,f(u))=\varnothing$, and $f$ isn’t a surjection.

• If $u\in L$, then $f(u)<0$, but $f(x)>0$ for every $x>u$, so $f[\mathbb{R}]\cap (f(u),0)=\varnothing$, and again $f$ is not a surjection.

In either case we have a contradiction, so $f$ cannot be a surjection.

If $f$ were a decreasing surjection, $-f$ would be an increasing surjection, so a decreasing function from $\mathbb{R}$ to $\mathbb{R}\setminus\mathbb{Q}$ can’t be a surjection either.

Note first that the OP is probably using "range" in the sense of "image", not in the sense of "codomain".

I suggest you use Theorem 119 in $\S$6.3 of these notes on discontinuities of weakly monotone functions. Here is the basic idea (I will assume $f$ is weakly increasing).

Case 1: The function is everywhere continuous. Then its image $f(\mathbb{R})$ must be an interval, by the Intermediate Value Theorem, so it certainly can't be $\mathbb{R} \setminus \mathbb{Q}$.

Case 2: The function is discontinuous at at least one point $a \in \mathbb{R}$. Here's where Theorem 119 comes in: then we have either $f(a^-) = \lim_{x \rightarrow a^-} f(x) < f(a)$ and the image contains no values in $(f(a^-),f(a))$ or $f(a) < f(a^+) = \lim_{x \rightarrow a^+} f(x)$ and the image contains no values in $(f(a),f(a^+))$. But every nonempty open interval contains irrational numbers.