Continuous bijection on open interval is homeomorphism

It is easy to prove by the intermediate value theorem that any continuous injection from an interval to $\mathbb{R}$ must be monotone (for instance, if $c<d<e$ and $f(c)<f(e)<f(d)$ then $f(e')=f(e)$ for some $e'\in (c,d)$; other failures of monotonicity can be handled similarly). Thus $f$ is either an order-preserving bijection or an order-reversing bijection $(a,b)\to\mathbb{R}$, which maps open intervals to open intervals and is hence a homeomorphism.

Another possibility is is to use the fact that the continuous image of a compact set is compact. If $C\subset (a,b)$ is closed, then it compact since it is bounded. It follows that $f(C)$ is compact, hence closed and bounded. Since $f$ sends closed sets to closed sets (and it is bijective), it also sends open sets to open sets. Hence, for every $U$ open in $(a,b)$, $f(U)$ is open. Then $(f^{-1})^{-1}(U)=f(U)$ is open and $f^{-1}$ is continuous.

There is an elementary, general, textbook result which answers your question, and which relies on the concept of a proper map $f : X \to Y$, meaning that the pre-image of every compact subset of $Y$ is compact in $X$.

Theorem: If $f : X \to Y$ is a continuous, proper bijection and $Y$ is a compactly generated Hausdorff space then $f$ is a closed map and hence a homeomorphism.

One might not prefer this as a proof for your specific problem, though, because one still needs to verify properness of your map $f : (a,b) \to \mathbb{R}$, and my guess is that any argument for properness would be similar to one of the existing answers by others.

This theorem is an exercise, for example, in Munkres' topology book.