Group representations and active/passive transformations

Active vs. Passive.

An active transformation is a map $T\colon \mathrm O(n)\to \mathrm{End}(\mathscr C(\mathbb R^n))$ defined as $$ (T_R\phi)(x):=\phi(R^{-1}x)\tag1 $$

A passive transformation is a map $f\colon\mathrm O(n)\to \mathrm{End}(\mathbb R^n)$ defined as $$ f_R(x):=Rx\tag2 $$

These two view points are related through $$ T_R\phi\equiv\phi\circ f_R^{-1}\tag3 $$

They both furnish representations of $\mathrm O(n)$: $$ T_R\circ T_{R'}=T_{RR'},\qquad f_R\circ f_{R'}=f_{RR'}\tag4 $$ as is trivially checked: $$ \begin{aligned} (T_R(T_{R'}\phi))(x)&\overset{(1)}=(T_{R'}\phi)(R^{-1}x)\overset{(1)}=\phi(R'^{-1}R^{-1}x)=\phi((RR')^{-1}x)\\ f_R(f_{R'}x)&\overset{(2)}=f_R(R'x)\overset{(2)}=RR'x \end{aligned}\tag5 $$ as required.

Needless to say, and because of equation $(3)$, establishing the group structure of $T$ automatically implies that of $f$ and vice-versa. We included both calculations to illustrate both viewpoints.

Left vs. Right.

OP is actually asking about a non-standard notion of active vs. passive, that I will call left vs. right for lack of a better term. I will focus on $T$, but the discussion in terms of $f$ is analogous.

A left (active) transformation is a map $T\colon \mathrm O(n)\to \mathrm{End}(\mathscr C(\mathbb R^n))$ defined as $$ (T_R\phi)(x):=\phi(R^{-1}x)\tag6 $$

A right (active) transformation is a map $\tilde T\colon \mathrm O(n)\to \mathrm{End}(\mathscr C(\mathbb R^n))$ defined as $$ (\tilde T_R\phi)(x):=\phi(Rx)\tag7 $$

Unlike before, these two transformations are two different transformations, not two different points of view of the same transformation. That being said, they are related through $$ T=\tilde T\circ \mathrm{inv}\tag8 $$ where $\mathrm{inv}\colon R\mapsto R^{-1}$ is the inverse map.

As discussed before, $T$ is a $G$-homomorphism, where $G$ is the group $G=(\mathrm{End}(\mathscr C(\mathbb R^n)),\circ)$. In other words, $T$ is a representation of $\mathrm O(\mathbb R^n)$, with $\mathrm{End}(\mathscr C(\mathbb R^n))$ being the representation space. The group product is just composition.

On the other hand, $\tilde T$ is a $G$-anti-homomorphism (indeed, $\tilde T$ is the composition of a homomorphism $T$ and the standard (involutive) anti-homomorphism $\mathrm{inv}$, cf. $(8)$). In other words, $\tilde G$ it is not quite a representation, in accordance with OP's claims. One should point out that $\tilde T$ is a $\tilde G$-homomorphism, where $\tilde G$ is the group $\tilde G=(\mathrm{End}(\mathscr C(\mathbb R^n)),\star)$, where the group product is defined as $\star=\circ\otimes\tau$, where $\tau\colon A\otimes B\mapsto B\otimes A$ is the swap map. In other words, $\star$ is the product $$ A\star B:=B\circ A\tag9 $$

That $\tilde G$ is indeed a group is an easy exercise that is left to the reader. This group is typically known as the opposite group of $G$, and is denoted by $G^\mathrm{op}$.

The conclusion is now immediate: $\tilde T$ is also a representation of $\mathrm O(n)$, but the group product is not $\circ$, but $\star$ instead.

I think an "active" transformation is a left action, and a "passive" transformation is a right action.

Active transformation:

$$(R_1 \cdot (R_2 \cdot \phi)) (x) = (R_2 \cdot \phi) (R_1^{-1} x) = \phi(R_2^{-1} R_1^{-1} x) = ((R_1 R_2) \cdot \phi) (x)$$

Passive transformation:

$$(R_1 \cdot (R_2 \cdot \phi)) (x) = (R_2 \cdot \phi) (R_1 x) = \phi(R_2 R_1 x) = ((R_2 R_1) \cdot \phi) (x)$$

Usually physicists don't ever think about right actions, but mathematically there's no reason not to.