Generalizing the commutator and anti-commutator

This generalization is called a ternary commutator, and there are $n$-commutator generalizations, as explained here:

It should be noted that commutator has been generalized to congruences for results in universal algebra. Look up reviews of "Commutator Theory For Congruence Modular Varieties" by Freese and McKenzie. Many far-reaching results are obtained by considering a commutator operation on congruence lattices.