Fractional Linear Transformations and their matrix form

It's a 2-by-2 complex matrix. It defines a linear map from $\Bbb C^2$ to $\Bbb C^2$. Therefore it defines a map from the projective line $\Bbb P^1(\Bbb C)$ to itself. We can think of this projective line as $\Bbb C\cup\{\infty\}$. So $M$ induces a map from $\Bbb C\cup\{\infty\}$ to itself. That map is $T_M$.

A "fractional linear transformation" is not a special kind of "linear transformation", so you'll have to unlearn a little bit of linear algebra to understand this issue.

Let's take apart the definitions to see what a fractional linear transformation actually is.

Start with the matrix $M = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$.

When we think of $M$ as a "linear transformation", that means that $M$ is treated as a function $$L_M : \mathbb{R}^2 \to \mathbb{R}^2 $$ where the elements of the domain and range of $L_M$ are vectors --- more specifically, column vectors. As you say, this function is evaluated by matrix multiplication, where the four coefficients of the matrix $M$ are applied with the following effect: if $v = \begin{pmatrix} x \\ y \end{pmatrix}$ then $$L_M(v) = \begin{pmatrix} a & b \\ c & d \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} ax+by \\ cx+dy \end{pmatrix} $$

When we think of $M$ as a "fractional linear transformation", we do not think of it as a function where the elements of the domain and range are vectors. Instead, $M$ is treated as a function where the elements of the domain and range are complex numbers: $$T_M : \mathbb{C} \to \mathbb{C} $$ This function is not a linear transformation of any kind, and it is not evaluated by any kind of matrix multiplication (at least, not at first...). Instead, it is evaluted by this formula, where the four coefficients of the matrix are applied to a different effect: $$T_M(z) = \frac{az+b}{cz+d} $$ (One caveat: the domain and range should really be the extended complex numbers $\mathbb{C} \cup \{\infty\}$, with appropriate formulas to interpret "infinity arithmetic")

So, was this terminology just pulled out of a hat? Why did someone decide to call this a "fractional linear transformation"?

The reason is that you can break the evaluation of $T_M(z)$ into a multi-step process in which you take "the fraction of a linear transformation". What I really mean by these words is simply that you do this:

1. Write $z$ as a fraction: $z = \frac{u}{v}$. For example, you could choose $u=z$ and $v=1$, or you could choose $v$ to be any complex number whatsoever and take $u=zv$. The final outcome of this multi-step process will be well-defined independent of choice.
2. Treating the numerator and denominator separately, write them as components of a column matrix: $\begin{pmatrix} u \\ v \end{pmatrix}$.
3. Apply $M$ as a linear transformation, using this column matrix as the input, producing the output $$\begin{pmatrix} a & b \\ c & d \end{pmatrix} \begin{pmatrix} u \\ v \end{pmatrix} = \begin{pmatrix} au+bv \\ cu+dv \end{pmatrix} $$ (Another caveat: $M$ is being treated as a linear transformation over the complex numbers, instead of just over the real numbers.)
4. Treat the the components of the output column matrix as a fraction and then do a little fraction arithmetic: $$\frac{au+bv}{cu+dv} = \frac{au+bv}{cu+dv} \cdot \frac{1/v}{1/v} = \frac{a\frac{u}{v} + b}{c \frac{u}{v} + d} = \frac{az+b}{cz+d} $$ Voila!