Cross product in complex vector spaces

For finding the correct definition to apply, one needs to know whether the scalar product is taken to be anti-linear in its first or its second argument. Assuming the first convention, the relation one would want to preserve for $\vec x=(x_1,x_2,x_3)$ and similarly for $\vec y, \vec z$, that one still has $$ (\vec x \times \vec y)\cdot\vec z= \left|\begin{matrix}x_1&y_1&z_1\\x_2&y_2&z_2\\x_3&y_3&z_3\\\end{matrix} \right|. $$ Note that the determinant is linear in all of its columns, so the left hand side needs to be an expression that is linear in the vector that appears directly as a column, which explains that one cannot use $\vec x\cdot(\vec y\times\vec z)$ instead, which is anti-linear in $\vec x$. Now it is easy to see that the coordinates of $\vec x \times \vec y$ should be taken to be the complex conjugates of the expressions in their usual definition, for instance $\overline{x_2y_3-x_3y_2}$ for the first coordinate.

One actually arrives at the same conclusion for a scalar product that is defined to be anti-linear in its second argument. However the identity that leads to this definition is different, namely the one which equates $\vec x\cdot(\vec y\times\vec z)$ to the above determinant.

Yes, this is correct definition. If $v$, $w$ are perpendicular vectors in $\Bbb C^3$ (according to hermitian product) then $v,w,v\times w$ form matrix in $SU_3$.

We can define complex cross product using octonion multiplication (and vice versa). Let's use Cayley-Dickson formula twice: $$(a+b^\iota)(c+d^\iota)=ac-\bar db+(b\bar c+da)^\iota$$ for quaternions $a,b,c,d$. Next set $a=u\mathbf j, b=v+w\mathbf j, c=x\mathbf j, d=y+z\mathbf j$ for complex numbers $u,v,w,x,y,z$. Then we obtain from above formula $$-u\bar x-v\bar y-\bar w z+(\bar vz-w\bar y)\mathbf j+[w\bar x-\bar uz+(-vx+uy)\mathbf j]^\iota$$ Applying complex conjugation to third complex coordinate we obtain formula for cross product. The first term is hermitian product of the vectors $(u,v,w)$, $(x,y,z)$. $$\begin {bmatrix}u\\ v \\ w \end{bmatrix}\times \begin {bmatrix}x\\ y \\ z \end{bmatrix}=\begin {bmatrix}\overline {vz}-\overline{wy}\\ \overline{wx}-\overline{uz} \\ \overline{uy}-\overline{vx} \end{bmatrix}$$

$SU_3$ is subgroup of octonion automorphism group $G_2$. Any automorphism of octonions can be obtained by fixing unit vector $\mathbf i$ on imaginary sphere $S^6$. It defines complex structure on perpendicular space $R^6$ via multiplication. Now in this complex structure any $SU_3$ element is octonion automorphism. So $G_2$ is fiber bundle $S^6 \times \times SU_3$.

Now going to "vice versa". Let's define octonions as pairs $(a,\mathbf v)$ where $a$ is complex number and $\mathbf v$ vector in $\Bbb C^3$. Then octonion multiplication can be defined as $$(a,\mathbf v)(b,\mathbf w)=(ab-\mathbf {v\cdot w},a\mathbf w+b \mathbf v + \mathbf {v \times w})$$ I hope that above argument with double Cayley-Dickson formula can be used to prove it although I have not done myself this calculation. The reader is urged to do it as an exercise :)

We can extend the definition of cross product to quaternions the same way. Extending it to octonions we need to be more careful. Freudenthal has done this using 3x3 matrices over octonions - so called Jordan algebra. Some kind of "cross product" is present in all exceptional Lie groups $F_4$, $E_6$, $E_7$, $E_8$ as these groups are called by Rosenfeld as automorphism groups of 2-dimensional projective planes over $\Bbb {O,C\otimes O, H \otimes O, O \otimes O}$. Have I flied away too far from original question ?

Physicists tend to use the (nonconjugated) "real formulae" for both dot product and cross products. So physicists' dot product equals $\vec x\cdot \vec y = \vec x^T \vec y$, although their inner product is $\langle\vec x,\vec y\rangle=\vec x^H \vec y$, where $\vec x^H = \mathrm{conj}(\vec x)^T$ (the transpose with complex-conjugated elements).

Usually physicists write $\vec x^\dagger$ instead of $\vec x^H$, whereas mathematicians may write $\vec x^*$ for $\vec x^H$ (alternatively, for the conjugate of $\vec x$, which is why I use here the Hermitian notation $\vec x^H$, to avoid ambiguity).

Non-conjugated dot and cross products are the standard practice in, e.g., almost all books on electromagnetism. Note also that physicists, modern mathematicians and almost everyone else tend to conjugate the first element in their inner products, as above, except that old-school mathematicians more often the second element, according to Wikipedia, Note 2: https://en.wikipedia.org/wiki/Inner_product_space#Notes

This way, physicists get the following formula: $$ (\vec x \times \vec y)\cdot \vec z= \left|\begin{matrix}x_1&y_1&z_1\\x_2&y_2&z_2\\x_3&y_3&z_3\\\end{matrix} \right|. $$

Mathematicians tend to have dot product equal to inner product, i.e., $\vec x\cdot \vec y = \langle\vec x,\vec y\rangle=\vec x^H \vec y$. Old-school mathematicians have $=\vec y^H \vec x$. (Note the old-school jumping order that makes their dot products linear with respect to their first arguments and conjugate-linear w.r.t. the second ones.)

Therefore, mathematicians tend to define cross$(x,y)=$ conj(physicscross$(x,y))$, i.e., $$ \vec x \times \vec y = \left[\begin{matrix}\overline{x_2y_3-x_3y_2}\\ \overline{x_3y_1 - x_1y_3}\\ \overline{x_1y_2-x_2y_1}\\ \end{matrix} \right] $$ (the overline symbols denote complex conjugation; remove them for physicists' cross product) to preserve the formula $$ (\vec x \times \vec y)\cdot\vec z= \left|\begin{matrix}x_1&y_1&z_1\\x_2&y_2&z_2\\x_3&y_3&z_3\\\end{matrix} \right|. $$ Old-school mathematicians have, instead, $\vec x \cdot (\vec y\times\vec z)=|\cdots|$, as their dot product conjugates the latter element, thus canceling the conjugation in the cross product this way.

The physicists' (non-conjugated) formulae can be found, e.g., here (gated): https://www.tandfonline.com/doi/pdf/10.1080/002073999287815

Matlab, Maple and Mathematica

Matlab uses modern mathematicians' formulae for the dot product, i.e., it conjugates the first element: dot$(x,y)=x^Hy$ (link https://se.mathworks.com/help/matlab/ref/dot.html); but it uses the physicists' (non-conjugated) formulae for the cross product: cross$(x,y)$ doesn't involve the conjugation of the output.

Thus, in Matlab, dot(i,1) = -i, cross([0,i,0],[0,0,1]) = (+i, 0, 0).
Therefore, $(\vec x \times \vec y)\cdot\vec z \ne |\cdots|$ in Matlab.

Mathematica conjugates neither. Maple conjugates the dot product (though you have the option to nonconjugate it) but not the cross product. https://www.maplesoft.com/support/help/Maple/view.aspx?path=LinearAlgebra/DotProduct https://www.maplesoft.com/support/help/Maple/view.aspx?path=LinearAlgebra/CrossProduct

My guess is that Scilab does not conjugate. Somebody hopefully certifies/corrects this paragraph.

This answer builds on the shoulders of the answer by Marc van Leeuwen on this same page.