Emergence of the orthogonal group

Your quote about Cartan thinking of $B_n$ and $D_n$ as 'projective groups..." is actually Cartan describing the lowest dimensional homogeneous space of these groups (except, of course, for a few exceptional cases such as $D_2$, which is not simple, and therefore should be left out of the description).

If you go just a little bit further in Cartan's 1894 Thesis, to Chapitre VIII, Section 9, you'll see that Cartan describes linear representations as well. For example, of $B_\ell$, he writes "C'est le plus grand groupe linéare et homogéne de l'espace à $2\ell{+}1$ dimensions qui laisse invariante la forme quadratique $$ {x_0}^2 + 2x_1x_{1'} +2x_2x_{2'} + \cdots + 2x_\ell x_{\ell'}" $$ with a similar description for $D_\ell$.

In fact, he gives the lowest dimensional representation of each of the simple groups over $\mathbb{C}$, including the exceptional ones and, except for $\mathrm{E}_8$, he explicitly describes the equations that define the representation. For example, he writes down an explicit homogeneous cubic in 27 variables and states that $\mathrm{E}_6$ is the the subgroup of $\mathrm{GL}(27,\mathbb{C})$ that preserves this cubic form.

For the summary theorem on the linear representations, see Chapitre VIII, Section 10, where he lists each of the lowest representations and notes the various low dimensional exceptional isomorphisms as well.

Remark 1: Cartan continues to refer to groups of type $B$ and $D$ merely as "the largest groups preserving a quadratic form in $n$ variables" or similar terms for a long time. Even in his papers of 1913–1915 classifying the real forms of the complex simple Lie groups, he uses such terminology, though he clearly finds the special case of the compact real forms of special interest.

The first place that Cartan actually refers to 'orthogonal groups' that I can recall are in his 1926–27 papers on the classification of Riemannian symmetric spaces. There, he begins referring to any subgroup of $\mathrm{GL}(n,\mathbb{R})$ that preserves a quadratic form as 'an orthogonal group' and then, later, finally refers to the maximal group that preserves a positive definite quadratic form as 'the orthogonal group'. I don't recall when or whether he used any notation such as $\mathrm{O}(n)$ or $\mathrm{SO}(n)$.

Whether the term 'orthogonal group' was original to him, I can't say. By that time, of course, Weyl had already started his research on compact Lie groups, and it may be that Weyl had already used the term 'orthogonal group' well before Cartan.

Remark 2: Euler's article (Problema algebraicum ob affectiones prorus singulares memorabile, Novi commentarii academiae scientiarum Petropolitanae 15 (1770) 1771, 75–106) discusses the problem of parametrizing the solutions of the equation $A^TA = I_n$ where $A$ is an $n$-by-$n$ matrix for $n=3$, $4$, and $5$, particularly the rational solutions. He does not use the terminology 'orthogonal' or 'group'. Nevertheless his article does contain some remarkable formulae that clearly anticipate the development of the algebra of quaternions.

For example, identifying $\mathbb{R}^4$ with the quaternions $\mathbb{H}$ in the usual way, it is a now-standard fact that every special orthogonal linear transformation $M$ of $\mathbb{R}^4=\mathbb{H}$ can be written, using quaternion multiplication, in the form $M(X) = A\,X\,\bar B$ where $A$ and $B$ are unit quaternions and $X\in\mathbb{H}$. (This is now the usual way that the double cover $\mathrm{Spin}(3)\times\mathrm{Spin}(3)\to\mathrm{SO}(4)$ is introduced.) Meanwhile conjugation $c:\mathbb{H}\to\mathbb{H}$ is orthogonal but has determinant $-1$, so every element of the non-identity component of $\mathrm{O}(4)$ can be written as $$M'(X) = Ac(X)\bar B = A\,\bar X\, \bar B = A\overline{BX} = Ac(BX).$$ Remarkably, Euler gives this formula for parametrizing $\mathrm{O}(4)$ in the form of the product of matrices $L_A\,c\,L_B$ (where $L_P$ denotes left multiplication by the quaternion $P$), many years before the 'official' discovery of quaternions.

There may be an earlier source, but Adolf Hurwitz 1897 is one upper bound:

A. Hurwitz, Über die Erzeugung der Invarianten durch Integration, Nachr. Ges. Wiss. Göttingen (1897), 71–90.

Hurwitz’s paper introduced and developed the notion of an invariant measure for the matrix groups SO(N) and U(N). He also specified a calculus from which the explicit form of these measures could be computed in terms of an appropriate parametrisation — Hurwitz chose to use Euler angles. This enabled him to define and compute invariant group integrals over SO(N) and U(N).

source: A. Hurwitz and the origins of random matrix theory in mathematics