A name for this property?
In noncommutative ring theory (which might be applied to real matrices), a map $\phi:R\to R$ such that $\phi(ab)=\phi(b)\phi(a)$ and $\phi(a+b)=\phi(a)+\phi(b)$ may be called a ring-homomorphism $R\to R^{\operatorname{op}}$.
What's $R^{\operatorname{op}}$? Basically if $(R,+,\cdot)$ is a ring, $R^{\operatorname{op}}$ is the ring $(R,+,\cdot^{\operatorname{op}})$ given by $a\cdot^{\operatorname{op}}b:=b\cdot a$, while the sum remains the same.
I'm not sure if there's a general name for this (other than "order-reversing homomorphism", or "a homomorphism $R \to R^{op}$", as said in the answer of G. Sassatelli) but many operations of this nature (including both of the ones you mention) are involutions.
Other examples include:
- The operation of inversion in any ring: if $a$ and $b$ are invertible elements, then $(ab)^{-1}=b^{-1}a^{-1}$.
- The operation of transposition on matrices: if $M$ and $N$ are any two matrices for which the product $MN$ is defined, then $(MN)^T = N^T M^T$.
Note that any involution necessarily takes the identity to itself: $M^* = (1 \cdot M)^* = M^* \cdot 1^* $, hence $1^*=1$.
Edited to add: Just to clarify, not all involutions have the desired property; as Dmitry Rubanovich points out in the comments, an involution does not necessarily reverse order (although most of the interesting ones do). And conversely not every order-reversing homomorphism is an involution; involutions are all bijective and satisfy $x^{**}=x$, which need not be the case for an arbitrary order-reversing homomorphism. But (as I wrote originally) many operations of this nature -- including both of the examples in the OP -- are involutions, and *-algebras (per Federico Poloni's comment) provide a rich source of additional examples.
Anti- (automorphism | endomorphism | homomorphism | isomorphism) of (groups | rings | monoids | semigroups | algebras... ).
Sample sentences:
The inversion operation in a group is an antiautomorphism of that group. Matrix transpose is an antiautomorphisms of the ring of matrices. Conjugation is an antiautomorphism of quaternions and octonions. [ More at https://en.wikipedia.org/wiki/Antihomomorphism ]
Those are the most common examples and are also involutions (performing operation twice returns to initial state).
It is more common to say anti*morphism or order-reversing morphism, than to talk about a morphism to the op-structure on the same object.