What did Frobenius prove about $M_{12}$?

It seems to me that Frobenius is using lots of specific facts about the permutation group $M_{12}$ (and $M_{24}$, respectively) here, and that he has no doubts about the existence of these groups. In particular, he is using specific subgroups of $M_{12}$ and $M_{24}$ (and then using induced characters). Most statements are given without proof, and so the reader would have to verify them by whatever tedious calculations (?).

The first paragraph of §5 reads:

By using the developed theorems, I have calculated the characters of all multiply transitive groups of degree $\leq 24$. Except for the symmetric and alternating groups of the various degrees, no group is known which is more than fivefold transitive, and only two five-transitive groups are known, both discovered by Mathieu, whose characters I want to give here.

In the next paragraph, Frobenius describes the partition of $M_{12}$ into conjugacy classes, without proofs or references, so as if this is generally known or easy to reproduce. He also explains notation in the character table, e.g., $(6)(3)(2)$ denotes the class of elements with cycle lengths $6,3,2,1$, and $(3)^4$ denotes the class of elements which decompose into $4$ cycles of length $3$. (In the linked scan of the paper, the first column of the table denoting the conjugacy classes is partly hidden in the binding.) The third paragraph of §5 explains more notation (second column contains centralizer orders, first row degrees of characters with superscript if there is more than one of the same degree).
The bottom paragraph on p. 567 reads:

By the theorems we have proved, $M_{12}$ has the characters $\alpha-1 : 11^{(1)}$, $\frac{1}{2}(\alpha-1)(\alpha-2)-\beta : 55^{(1)}$ and $\frac{1}{2}\alpha(\alpha-3) + \beta : 54$. [Here, $\alpha$ denotes the number of elements fixed and $\beta$ the number of $2$-cycles.] Formula (5), §2 contains three characters of the third dimension of the symmetric group $S_{12}$, and each of these decomposes into two characters of $M_{12}$, namely $99+55^{(3)}$, $120+45$, $144+176$. The first two characters of Formula (6), §2 decompose into $11^{(2)} + 54 + 66 + 144 $ and $ 66 + 120 + 144 $.

Paragraph following the table on p.568:

The substitutions of $M_{12}$ fixing a symbol constitute a $4$-transitive group $M_{11}$ of degree $11$ and order $11.10.9.8$. Furthermore, $M_{12}$ contains a subgroup which is isomorphic to $M_{11}$, is $3$-transitive of degree $12$ and has order $12.11.10.6$. So one can also use this group to represent $M_{12}$ as a transitive permutation group on $12$ symbols, and thus one obtains an outer automorphism of $M_{12}$, and thereby the classes $(8)(2)$ and $(8)(4)$ and their squares $(4)^2$ and $(4)^2(2)^2$ are interchanged. By this automorphism, the character $11^{(1)}$ is sent to $11^{(2)}$ and the character $55^{(1)}$ is sent to $55^{(2)}$. The other characters are computed using the subgroup $M_{11}$.

Next paragraph:

With benefit, one can also use the following remarkable subgroup of $M_{12}$: Let $(1,2,3,4,5,6)(7,8,9)(10,11)(12)$ be a substitution in the class $(6)(3)(2)$. Then all substitutions $R$ in $M_{12}$, which move the first $6$ (and the last $6$) only among them, constitute a group of order $6!$. Each such substitution $R$ decomposes into two substitutions $R_1$, only permuting the first $6$, and $R_2$, only permuting the last $6$ symbols among themselfs. Both $R_1$ and $R_2$ run through the $6!$ substitutions of the symmetric group $S_6$ of degree $6$, and $R_1$ and $R_2$ correspond by the well known outer autmorphism of this group. In fact, the classes $(6)$, $(3)^2$ and $(2)^3$ of $S_6$ thus correspond to the classes $(3)(2)$, $(3)$ and $(2)$, and their union gives the classes $(6)(3)(2)$, $(3)^3$ and $(2)^4$ of $M_{12}$. The principal character of $S_6$ gives the reducible character $1 + 11^{(1)} + 11^{(2)} +54 +55^{(3)}$ of $M_{12}$, the other linear character of $S_6$ gives the character $11^{(2)} + 55^{(2)} +66$, and thus one gets the two simple characters $55^{(3)}$ and $66$. Then the above formulas give all characters except $16^{(1)}$ and $16^{(2)}$, which can easily be determined from the bilinear relations [=orthogonality relations?].

Last paragraph of §5 (p. 569):

If $\chi(R)$ is a character of $H$, then
$$ \frac{1}{2}(\chi(R)^2 - \chi(R^2)) \:,\quad \frac{1}{2}(\chi(R)^2 + \chi(R^2) ) \tag{1.} $$ are linear combinations of characters with positive integer coefficients. If one chooses for $\chi(R)$ the character $16^{(1)}$, then one gets $120$ and $16^{(2)} + 54 + 66$.

Section 6 on $M_{24}$ is even more sketchy, but has roughly the same structure. Frobenius tells us that he used two other subgroups in addtion to the stabilizer $M_{23}$. One is the setwise stabilizer $M_{16+8}$ of a subset of order $16$, which is the $15+1$-part of an element of cycle structure $15+1+5+3$. The permutations induced on the $16$-subset are said to form the $3$-transitive linear group of order $2^4(2^4-1)(2^4-2)(2^4-2^2)(2^4-2^3)$, and the permutation group induced on the $8$-set is $A_8$.
According to the bottom paragraph on p.~570, on gets another subgroup by taking the stabilizer of a $12$-subset. The stabilizer is isomorphic to $M_{12}$. Each element of the stabilizer decomposes into two permutations $R_1$ and $R_2$ on a $12$-subset and its complement, the correspondence between $R_1$ and $R_2$ is given by the outer automorphism of $M_{12}$ described in §5. Unfortunately, Frobenius gives us no clue how to choose that $12$-subset ("in passender Art")!

This summary of Frobenius' 1904 paper might be of use:

Thomas Hawkins, The Mathematics of Frobenius in Context (page 527-528).

Not really an answer, and probably contained in Frobenius's paper, but a starting point might be that (using a result now usually credited to Blichfeldt) if $G$ is a sharply $5$-transitive group (of any degree) with permutation character $\chi$, and trivial character $\chi_{0},$ then $\chi(\chi - \chi_{0})(\chi-2\chi_{0})(\chi- 3\chi_{0})(\chi-4\chi_{0})$ must be the regular character ( note that the given product of virtual characters vanishes on all non-identity elements of $G$ and takes value $|G|$ on the identity). Hence decomposing each of $\chi,\chi^{2},\chi^{3},\chi^{4},\chi^{5}$ into sums of irreducible characters would yield all non-trivial irreducible characters of $G$ ( and more besides).