css3 rotate transition, doesn't take shortest way

The transform is doing exactly what you tell it to.

It starts at 359deg and goes to 1deg. You are looking to 'rollover' 360deg back to 1deg, which is really 361deg. The way the transform transitions work is that it interpolates between values.

The solution to your problem is to make a counter variable that holds the degrees of rotation:

var rot = 0;  // lets start at zero, you can apply whatever later

To apply a rotation, change value:

rot = 359;
// note the extra brackets to ensure the expression is evaluated before
//   the string is assigned this is require in some browsers
element.style.transform = ("rotate( " + rot + "deg )");

so if you do this:

rot = 1;
element.style.transform = ("rotate( " + rot + "deg )");

it goes back. So you need to see if it is closer to 360 or 0 regardless how many rotations it has been through. You do this by checking the value of element.style.transform which is just the current rot value and then comparing to the new rot value. However, you need to do this with respect to how many rotations may exist, so:

var apparentRot = rot % 360;

Now no matter how many spins it has had, you know how far around it is, negative values are equal to the value + 360:

if ( apparentRot < 0 ) { apparentRot += 360; } 

Now you have normalized any negative values and can ask whether a positive rotation (through 360deg in your case) or negative is needed. Since you seem to be giving the new rotation value as 0-360deg, this simplifies your problem. You can ask if the new rotation + 360 is closer to the old value than the new rotation itself:

var aR,          // what the current rotation appears to be (apparentRot shortened)
    nR,          // the new rotation desired (newRot)
    rot;         // what the current rotation is and thus the 'counter'

// there are two interesting events where you have to rotate through 0/360
//   the first is when the original rotation is less than 180 and the new one
//   is greater than 180deg larger, then we go through the apparent 0 to 359...
if ( aR < 180 && (nR > (aR + 180)) ) {
    // rotate back
    rot -= 360;
} 

//   the second case is when the original rotation is over 180deg and the new
//   rotation is less than 180deg smaller
if ( aR >= 180 && (nR <= (aR - 180)) ) {
    // rotate forward
    rot += 360;
}

Other than this, simply adding the value of the new rotation to rot is all that is needed:

rot += (nR - aR); //  if the apparent rotation is bigger, then the difference is
                  //  'negatively' added to the counter, so the counter is
                  //  correctly kept, same for nR being larger, the difference is
                  //  added to the counter

Clean it up a bit:

var el, rot;

function rotateThis(element, nR) {
    var aR;
    rot = rot || 0; // if rot undefined or 0, make 0, else rot
    aR = rot % 360;
    if ( aR < 0 ) { aR += 360; }
    if ( aR < 180 && (nR > (aR + 180)) ) { rot -= 360; }
    if ( aR >= 180 && (nR <= (aR - 180)) ) { rot += 360; }
    rot += (nR - aR);
    element.style.transform = ("rotate( " + rot + "deg )");
}

// this is how to intialize  and apply 0
el = document.getElementById("elementYouWantToUse");
rotateThis(el, 0);

// now call function
rotateThis(el, 359);
rotateThis(el, 1);

The counter can go positive or negative, it doesn't matter, just use a value between 0-359 for the new rotation.

As you mention, the CSS transition does not find the shortest path between the starting and ending rotations, but animates fully between the starting and ending rotation. For example, going from 0 to 720 will cause the compass to animate spinning twice, rather than standing still as you desire.

The solution is on each update to set the rotation angle to the closest equivalent angle to the current rotation angle.

The function to calculate this angle needs to handle large angles and negative angles, since the compass can easily spin multiple time in either direction.

The trick is to calculate the difference between the old and requested angles and transform it into the range (-180, 180) so that the animation always takes the shortest path in the correct direction. You simply add this difference to the old angle to get the new angle.

function closestEquivalentAngle(from, to) {
    var delta = ((((to - from) % 360) + 540) % 360) - 180;
    return from + delta;
}

The above function is pretty simple aside from clamping the difference to (-180, 180). It makes use of modulus arithmetic to do this:

1. The difference angle is just an angle, so we don't take into account from or to anymore
2. Use the remainder operator to get the angle in (-360, 360); a gotcha: remainder is not the same as mathematical modulus for negative numbers
3. Add 540, this shifts the angle to (180, 900); this gets rid of the negative values and shifts the angle by 180 degrees
4. Take the remainder again to get our 180-shifted value to (0, 360); this is equivalent to mathematical modulus since we have eliminated negative values
5. Subtract 180 to unshift the value to the original angle in (-180, 180)