In mathematics, which word should I use, "in" or "on" or "over"?

My first thought is that regardless of choice, people will know what you mean. However, there are a few things to consider. To me, to say in is not the most natural of usage, though I wouldn't label it as inaccurate. Over, similarly, works but perhaps isn't the most natural choice. As Wesley mentioned, it makes me think of a vector space over a field. Had you not asked, and had I simply been talking, I have every expectation I would have said on. There is one small caveat:

When you speak of a function $f$ mapping $\mathbb{R}^3$ onto its domain, you are indicating the mapping is surjective. Therefore there could be an implicit assumption here (should you say that $f$ is bounded on $\mathbb{R}^3$) that your $f$ is also surjective. However, $f$ is defined on its domain so this usage is certainly valid.

Side note: if you like to be particular with your written expression, one further comment is that I think it is likely you mean that (and indeed you'll note I said that) $f$ is bounded on\over\in $\mathbb{R}^3$, which refers to the function, and not $f(x)$ which refers to the value of the function evaluated at $x$. It's worth noting that this is not a common distinction (in my experience) that is made outside of a pure analysis setting.