Enumerating cosets of the modular group

Since, as you say, you are talking about "fractional linear transformations not matrices", you are really interested in the group $PSL(2,\mathbb Z)$ rather than $SL(2,\mathbb Z)$. As it has already been pointed out, it is isomorphic to the free product of cyclic groups of orders 2 and 3. Let me denote the generator of order 2 by $a$, and the generator of order 3 by $b$. In these terms you are looking for a transversal (a system of coset representatives) to the cyclic group generated by $ab$. The simplest is the transversal which consists of $e,a$ and all words in the free product which begin with $b$ (i.e., all words of the form $bab^{\epsilon_1}ab^{\epsilon_2}\dots$ with $\epsilon_i=1,2$).

The specific question you asked has been answered, but you asked for an algorithm, and there does exist an implemented algorithm that can solve problems of this type.

Under certain conditions, which are roughly equivalent to $H$ being a finitely generated quasiconvex subgroup of an automatic group $G$, you can construct a "coset word acceptor" as part of a "coset automatic structure" for $H$ in $G$, where $G$ is defined by a finite presentation, and $H$ is defined by a finite set of generating words. The coset word acceptor is a finite state automaton which accepts a unique word over the generators of $G$ for each right coset of $H$ in $G$. In particular, this will work for any finitely generated subgroup of a virtually free group, as in your example.

The algorithm is implemented as part of the $\mathtt{kbmag}$ package, which is available as a standalone or as a $\mathsf{GAP}$ package. With your example, the coset word acceptor has $6$ states.