Showing any countable, dense, linear ordering is isomorphic to a subset of $\mathbb{Q}$

In fact, there is no need to assume that $R$ is dense. The fact is that every countable linear order embeds into the rational line. Another way to say this is that the rational line (or any countable dense linear order) is universal for countable linear orders.

The proof is just as you describe. Given a countable order $\langle A,R\rangle$, enumerate $A=\{a_0,a_1,\ldots\}$ and build the full isomorphism of $A$ into $\mathbb{Q}$ by successively defining $f(a_n)$. At any stage of the construction, we have a partial isomorphism defining the images of $a_i$ for $i\lt n$, and we want to define $f(a_n)$. The images $f(a_i)$ form a finite subset of $\mathbb{Q}$, which divides $\mathbb{Q}$ into finitely many intervals. So we look at how $a_n$ sits with respect to the $a_i$ for $i\lt n$, that is, at which interval in $A$ it sits in among those finitely many endpoints, and then choose any point in $\mathbb{Q}$ that fits in the corresponding interval in the image, and let that point be $f(a_n)$. Thus, we have made an order-preserving map at each stage, and so the full map is an order-isomorphism of $A$ with a subset of $\mathbb{Q}$ as desired.

Cantor proved also that in the case $R$ is dense and the order $\langle A,R\rangle$ has no endpoints, then in fact $\langle A,R\rangle$ is isomorphic to $\mathbb{Q}$, and one can prove this by using the back-and-forth technique, using the same idea, but now one must also ensure not only that the $n$-th point of the domain gets added to the isomorphism, but also that the $n$-th point of the range is added. To prove the property you requested, however, you only need the "forth" part of the back-and-forth construction.

what you are doing by building $f$ step-by-step is called building a partial map. Having already picked $b_{i}$ ($=f(a_{i})$) for $i<m$, you pick $b_{m}$ so that it preserves the relations that $a_{m}$ has with the $a_{i}$ for $i<m$. This is possible because $\mathbb{Q}$ is dense and has no endpoints. So, at every state you can enlarge the partial isomorphism. After $\omega$-many steps, the domain of the map will be all of $A$, and the $f$ you will have built will be an order preserving injection of $(A,R)$ into $\mathbb{Q}$.

To see that you can always extend the partial map, note that you only have to meet finitely many conditions each time. So your $a_{i}$ will atmost be between some two previous elements, or larger than all of them or greater than all of them. But since $\mathbb{Q}$ is dense and has no endpoints, in each case we can pick a corresponding $b_{i}$.

Also, notice that I made no assumptions here about $(A,R)$ here, except that it is a countable linear order. In case $(A,R)$ is countable, dense and has no endpoints, you can do even better, ie you can in a similar way build an isomorphism between $(A,R)$ and $(\mathbb{Q},<)$. The way to do this is similar to the above one, except you enumerate $A$ and $\mathbb{Q}$ and then build a isomorphism element by element such that it respects the orderings and has range $\mathbb{Q}$ and domain $A$. One way to do this is to first map an element of $A$ to an element of $\mathbb{Q}$ and then map the least unmapped element of $\mathbb{Q}$ in the enumeration to an element of $A$.

Keywords for what you are looking for are: Back-and-forth method, Ehrenfeucht-Fraïssé game. Any model theory text book (Hodges, Marker, Poizat etc) should have a detailed account of this method.

Your scheme is a simply an inductive (or recursive) definition of a function. To define a function $f : \mathbb N \to X$ (or any countable set as domain) according to this scheme you need i) supply the base case $f(1)$, and ii) for each $n > 1$ supply a rule how to compute $f(n)$ from its predecessors $f(n-1), \ldots, f(1)$.

In your example, let $A = \{a_1, a_2, a_3,\ldots \}$ be a countable infinite lineary ordered set, define $f(a_1) \in A$ arbitrary (for example $f(a_1) := 0$). For the induction step, suppose $f(a_1), \ldots, f(a_{n-1})$ have been defined ($n > 1$), then define $f(a_n)$ according to the following cases:

i) $a_n > a_k$ for all $k = 1,\ldots, n-1$, then set $f(a_n) = n$.

ii) $a_n < a_k$ for all $k = 1,\ldots, n-1$, then set $f(a_n) = -n$.

iii) $a_1 < \ldots < a_i < a_n < a_j < \ldots < a_n$, then set $$ f(a_n) = \frac{\phi(a_i) + \phi(a_j)}{2} $$ (or some other rational number between them).

This yield an order preserving injection into $\mathbb Q$ (and therefore an order preserving bijection to a subset $B \subseteq \mathbb Q$), note that the existence of the dense set $R$ was no used here. Also note that the injection isn't a surjection in general, and sometimes it is not possible to give one, for example it is not possible to find an order preserving bijection from $\mathbb Z$ to $\mathbb Q$.

So what is this dense set $R$ useful for? You need this if you want to find a surjection onto $\mathbb Q$, but as this is not clearly stated I think this exercise is not well-phrased. Anyway, I will state a result that might interest you, cause there something like your set $R$ is actually needed:

Theorem: Let $(A, \precsim)$ be a lineary ordered set, then the following two conditions are equivalent:

(i) There is a finite or countable order-dense subset of $A$,

(ii) There is an injection of $(A, \precsim)$ into $(\mathbb R, \le)$.

A proof of this could be found in:

Foundations of Measurement, Volume I, by D.H. Krantz, R.D. Luce, P. Suppes, A. Tversky.