Find the minimum of $P = (a - b)(b - c)(c - a)$

WLOG $a\ge b\ge c$ and let $x=a-b,y=b-c,z=c-a$

We observe $x+y+z=0$ with $x,y\ge 0$.and as you have found $x^2+y^2+z^2=12$

Elimination of $z$ results in: ${(x+y)}^2=6+xy ...(1)$

since ${(x+y)}^2\ge 4xy$ which means $0\le xy\le 2$

let $xy=t$

since $0\le t\le 2$

Now $x^2y^2z^2=t^2(6+t)\le 6.2^2+2^3=32$....(using (1) and $z=-(x+y)$)

or $|xyz|\le 4\sqrt{2}$

The inequality $$(a - b)^2 + (b - c)^2 + (c - a)^2 \geq 3 \sqrt[3]{((a - b)(b - c)(c - a))^2}$$ become equality when $a-b=b-c=c-a,$ or $a=b=c,$ but for this value then $$(a-b)(b-c)(c-a)=0 \ne -8.$$ This is my solution, we have $$P^2 = \frac{4(a^2+b^2+c^2-ab-bc-ca)^3-(a+b-2c)^2(b+c-2a)^2(c+a-2b)^2}{27} \quad (1)$$ $$\leqslant \frac{4}{27}(a^2+b^2+c^2-ab-bc-ca)^3.$$ Therefore $$P^2 \leqslant \frac{4}{27}(a^2+b^2+c^2-ab-bc-ca)^3 = 32,$$ or $$-4\sqrt 2 \leqslant (a-b)(b-c)(c-a) \leqslant 4\sqrt 2.$$ So $P_{\min} = -4\sqrt 2,$ equality occur when $a=1,\;b=1+2\sqrt 2,\;c=1+\sqrt 2.$

Note. How to find constant $\frac{4}{27}?$

For $(a-b)(b-c)(c-a) \ne 0,$ setting $x=a-b,\,y=b-c,$ then $$F = \frac{(a-b)^2(b-c)^2(c-a)^2}{(a^2+b^2+c^2-ab-bc-ca)^3} = \frac{x^2y^2(x+y)^2}{(x^2+xy+y^2)^3}.$$ From $x^2+xy+y^2 \geqslant \frac{3}{4}(x+y)^2$ and the AM-GM inequality, we have $$F \leqslant \frac{64}{27} \cdot \frac{x^2y^2}{(x+y)^4} \leqslant \frac{4}{27}.$$ From this proof we get $$\frac{4}{27}- \frac{x^2y^2(x+y)^2}{(x^2+xy+y^2)^3}=\frac{(x-y)^2(2y+x)^2(2x+y)^2}{(x^2+xy+y^2)^3}.$$ It's equivalent to the identity $(1).$

Using $x=a-b,y=b-c$, then $-(x+y)=c-a$, so $P=-xy(x+y)=-(x^2y+xy^2)$.

The condition $(a−b)^2+(b−c)^2+(c−a)^2=12$ becomes $g=x^2+y^2+xy=6$.

Now using Lagrange multipliers: \begin{align} \nabla P&=-\langle2xy+y^2,x^2+2xy\rangle\\ &=-\langle y(2x+y),x(x+2y)\rangle\\ \\ \nabla g&=\langle2x+y,x+2y\rangle \end{align}

Now letting $$\nabla P=\lambda\cdot\nabla g$$ $$\lambda=-y,\,\lambda=-x$$ So $x=y$, plugging this back in $g$: $$3x^2=6$$ $$x=\pm\sqrt2$$ Then the extreme values of $P$ are: $$P=-2\cdot\pm2\sqrt2=\pm4\sqrt2$$