If not associative, then what?

Is there any way to characterize all different possible types of operation which are not associative?

This is too broad and subjective to answer I think. What exactly is a "type" of operation? I assume you're already talking about binary operations, so presumably a "type" of operation is one which satisfies certain identities, like the associative identity. Certain specific examples come to mind:

• Jacobi identity for Lie algberas,
• Jordan identity for Jordan algebras,
• Moufang identites for loops,
• Self-distributive laws for racks and quandles,

and certainly others (I am no an expert in nonassociative algebra). Many of the identities above are not three-variable identities, but still. Generally, interesting algebras and their identities are not chosen randomly but rather follow from certain canonical examples whose properties are generalized. The algebras are meant to represent certain structures, and the identities ensure that. For instance, Lie algberas linearize Lie groups, and similarly Jordan algebras linearize projective spaces, Moufang identities generalize octonions' alternativity, racks and quandles represent how groups act on themselves by conjugation, etc.

Ultimately there is a "type" of operation for every possible set of "words" you can pick from the free magma (or if you allow addition, free nonassociative algebra) on so many generators. (There is going to be redundancy in this - different sets of words can yield the same class of algebras.)

Can we say that if ∗ is not associative, then it must instead satisfy one of set of other possible properties, depending on any other additional operations that we have on our set $X$?

Probably not. For instance the free nonassociative algebra on some generating set strikes me as a candidate for not having any "properties" (i.e. identities).

Is it ever useful to consider an 'associative commutator' for a given non-associative ∗?

Yes. The associator is useful for instance in (efficiently) proving the octonions are an alternative algebra (which is like halfway to being associative), which is in turn useful for many things like simplifying octonion expressions and classifying subalgebras and reasoning about automorphisms of $\mathbb{O}$. The octonion associator also gives rise to the exceptional ternary 8D cross product.

There's probably a lot more you can do with it in general nonassociative algebras but I wouldn't know.

Why do Lie algebras use this Jacobi identity

Consider where Lie algebras come from. Start with a Lie group $G$. The tangent space $\mathfrak{g}$ tells you all the directions that one-parameter subgroups can point in. The addition operation on $\mathfrak{g}$ corresponds to the group operation on $G$. Indeed, the exponential $\exp:\mathfrak{g}\to G$ is approximately linear in a neighborhood of $0$ with quadratic error term. As $G$ acts on itself by conjugation (and there are many sources listing example after example to show conjugation in a group is very important), so too it acts on $\mathfrak{g}$ by conjugation. Define $\mathrm{Ad}_A(X)=AYA^{-1}$ for $A\in G,Y\in\mathfrak{g}$. If we differentiate this at $A=I$ with tangent vector $X$ we get $\mathrm{ad}_X(Y)=XY-YX=[X,Y]$, the "commutator bracket." Note the adjoint action preserves this operation, and if we differentiate $\mathrm{Ad}_A[Y,Z]=[\mathrm{Ad}_AY,\mathrm{Ad}_AZ]$ at $A=I$ again with the product rule we get the identity $\mathrm{ad}_X[Y,Z]=[\mathrm{ad}_XY,Z]+[Y,\mathrm{ad}_XZ]$, which says $\mathrm{ad}_X$ is a "derivation" (i.e. satisfies the "product rule" like a derivative, but with the commutator bracket instead of multiplication). This identity may be rearranged to the more cyclically symmetric form you know as the Jordan identity.

All of the other identities I listed above have similar stories of where they come from. The Jordan identity comes from an algebraic investigation of spaces of Hermitian matrices (which are the span of projection operators, which correspond to points in projective spaces). Apparently the Jordan identity also has an interpretation in terms of the inversion symmetry of a Riemannian symmetric space, but I don't know how that story goes. The Moufang identity comes from investigating real normed division algebras, which leads to the octonions, which leads to the alternative identities, and then the simplest four-term identities one can check are where one term is repeated. The self-distributive law for racks and quandles comes from the fact conjugation is an automorphism in a group.

"What about operations which are not associative?" In many areas we encounter non-associative algebra structures, e.g., in operad theory, homology of partition sets, deformation theory, geometric structures on Lie groups, renormalisation theory in physics and many more.

In a certain sense one can answer your question what else can happen. One way is, to classify all nonassociative algebras defined by the action of invariant subspaces of the symmetric group $S_3$ on the associator of the considered laws, see for example here. But of course these are not all possibilities.

A well known example of a non-associative algebra structure related to Lie algebras are pre-Lie algebras (also called left-symmetric algebras). They satisfy the identity $$ (x,y,z)=(y,x,z) $$ for all $x,y,z\in A$, where $(x,y,z)$ is the associator. In particular, associative algebras are a trivial example where both sides are zero, i.e., with $0=0$. Then the commutator $$ [x,y]=xy-yx $$ is a Lie bracket, see Is there a relationship between associators and commutators?

Pre-Lie algebra arise in algebra, geometry and and physics, see my survey article here. They play an important role for crystallographic groups, fundamental groups of affinity flat manifolds (Milnor), Gerstenhaber deformation theory, Rota-Bater operators and Yang-Baxter equations, just to name a few key words.