Understanding Newton method when optimizing cost function depending on a triangle.
Why is the Newton method attempting to rotate instead of warping the triangle?
I'm not sure what you mean by "warping the triangle", but...
Clearly your cost function attains its local minima of $0$ at exactly the points $(x_1, x_2, x_3)$ for which the $x$ and $y$ components of $n$ are zero. So any solution necessarily will have all of $x_1, x_2, x_3$ all lying in some horizontal plane $z = C$. The Newton optimizer is simply converging to one of these solutions.
You of course have the issues that
- As indicated above, your optimum is (highly) non-unique. You could remedy this by introducing some sort of regularization, e.g. a similarity constraint as suggested by @Christian Blatter in the comments.
- Your cost function is not differentiable on the subspace $n=0$. It appears as if the Newton optimizer is currently converging to a valid minimum away from this subspace, but you would probably be safer using a globally differentiable cost function. Note that, alternatively, the above regularization suggestion would also remedy this issue by fixing $\| n \| > 0$.
but I believe that first paragraph answers your explicit question.