Direction of shortest rotation between two vectors
As you've written in your code, the angle between two (normalized) vectors is the inverse cosine of their dot product.
To get a signed angle, you can use a third vector representing the normal of the plane that the other two vectors lie on -- in your 2D case, this would be a 3D vector pointing straight "up", say (0, 0, 1).
Then, take the cross-product of the first vector (the one you want the angle to be relative to) with the second vector (note cross-product is not commutative). The sign of the angle should be the same as the sign of the dot product between the resulting vector and the plane normal.
In code (C#, sorry) -- note all vectors are assumed to be normalized:
public static double AngleTo(this Vector3 source, Vector3 dest)
{
if (source == dest) {
return 0;
}
double dot; Vector3.Dot(ref source, ref dest, out dot);
return Math.Acos(dot);
}
public static double SignedAngleTo(this Vector3 source, Vector3 dest, Vector3 planeNormal)
{
var angle = source.AngleTo(dest);
Vector3 cross; Vector3.Cross(ref source, ref dest, out cross);
double dot; Vector3.Dot(ref cross, ref planeNormal, out dot);
return dot < 0 ? -angle : angle;
}
This works by taking advantage of the fact that the cross product between two vectors yields a third vector which is perpendicular (normal) to the plane defined by the first two (so it's inherently a 3D operation). a x b
= -(b x a)
, so the vector will always be perpendicular to the plane, but on a different side depending on the (signed) angle between a
and b
(there's something called the right-hand rule).
So the cross product gives us a signed vector perpendicular to the plane which changes direction when the angle between the vectors passes 180°. If we know in advance a vector perpendicular to the plane which is pointing straight up, then we can tell whether the cross product is in the same direction as that plane normal or not by checking the sign of their dot product.