How to calculate rotation angle from rectangle points?

The center of the rectangle is right between two opposite corners:

cx = (x1 + x3) / 2
cy = (y1 + y3) / 2

The size of the rectangle is the distance between two points:

w = sqrt(pow(x2-x1, 2) + pow(y2-y1, 2))
h = sqrt(pow(x3-x2, 2) + pow(y3-y2, 2))

The corners of the gray rectangle can be calculated from the center and the size, for example the top left corner:

x = cx - w / 2
y = cy - h / 2

The angle is the arctangent of a side of the square:

a = arctan2(y4 - y1, x4 - x1)

(I'm not sure exactly which angle it returns, or what angle you expect for that matter, so you get to test a bit.)

You can use any coordinate pair on the same side to calculate the rotation angle. Note that mathematic angles normally assume 0 as long the +ve X axis and increase by rotating anti–clockwise (so along the +ve Y axis is 90°, -ve X axis is 180° and so on).

Also, javascript trigonometry functions return values in radians that must be converted to degrees before being used in a CSS transform.

If the shape is not rotated more than 90°, then life is fairly simple and you can use the tanget ratio of a right angle triangle:

tan(angle) = length of opposite side / length of adjacent side

For the OP, the best corners to use are 1 and 4 so that rotation is kept in the first quadrant and clockwise (per the draft CSS3 spec). In javascript terms:

var rotationRadians = Math.atan((x1 - x4) / (y1 - y4));

To convert to degrees:

var RAD2DEG = 180 / Math.PI;
var rotationDegrees = rotationRadians * RAD2DEG;

If the rotation is more than 90°, you will need to adjust the angle. e.g. where the angle is greater than 90° but less than 180°, you'll get a -ve result from the above and need to add 180°:

  rotationDegrees += 180;

Also, if you are using page dimentions, y coordinates increase going down the page, which is the opposite of the normal mathetmatic sense so you need to reverse the sense of y1 - y4 in the above.

Edit

Based on the orientation of points in the OP, the following is a general function to return the center and clockwise rotation of the rectangle in degrees. That's all you should need, though you can rotate the corners to be "level" yourself if you wish. You can apply trigonometric functions to calculate new corners or just do some averages (similar to Ian's answer).

/** General case solution for a rectangle
 *
 *  Given coordinages of [x1, y1, x2, y2, x3, y3, x4, y4]
 *  where the corners are:
 *            top left    : x1, y1
 *            top right   : x2, y2
 *            bottom right: x3, y3
 *            bottom left : x4, y4
 *
 *  The centre is the average top left and bottom right coords:
 *  center: (x1 + x3) / 2 and (y1 + y3) / 2
 *
 *  Clockwise rotation: Math.atan((x1 - x4)/(y1 - y4)) with
 *  adjustment for the quadrant the angle is in.
 *
 *  Note that if using page coordinates, y is +ve down the page which
 *  is the reverse of the mathematic sense so y page coordinages
 *  should be multiplied by -1 before being given to the function.
 *  (e.g. a page y of 400 should be -400).
 *
 * @see https://stackoverflow.com/a/13003782/938822
 */
function getRotation(coords) {
    // Get center as average of top left and bottom right
    var center = [(coords[0] + coords[4]) / 2,
                  (coords[1] + coords[5]) / 2];

    // Get differences top left minus bottom left
    var diffs = [coords[0] - coords[6], coords[1] - coords[7]];

    // Get rotation in degrees
    var rotation = Math.atan(diffs[0]/diffs[1]) * 180 / Math.PI;

    // Adjust for 2nd & 3rd quadrants, i.e. diff y is -ve.
    if (diffs[1] < 0) {
        rotation += 180;
      
    // Adjust for 4th quadrant
    // i.e. diff x is -ve, diff y is +ve
    } else if (diffs[0] < 0) {
        rotation += 360;
    }
    // return array of [[centerX, centerY], rotation];
    return [center, rotation];
}