Calculating vertices of a rotated rectangle

I would just transform each point, applying the same rotation matrix to each one. If it's a 2D planar rotation, it would look like this:

x' = x*cos(t) - y*sin(t)
y' = x*sin(t) + y*cos(t)

where (x, y) are the original points, (x', y') are the rotated coordinates, and t is the angle measured in radians from the x-axis. The rotation is counter-clockwise as written.

My recommendation would be to do it out on paper once. Draw a rectangle, calculate the new coordinates, and redraw the rectangle to satisfy yourself that it's correct before you code. Then use this example as a unit test to ensure that you coded it properly.

I think you were on the right track using atan() to return an angle. However you want to pass height divided by width instead of the other way around. That will give you the default (unrotated) angle to the upper-right vertex of the rectangle. You should be able to do the rest like this:

// Get the original/default vertex angles
GLfloat vertex1_theta = RAD_TO_DEG( atan(
            (m_height/2 * m_scaleY)
            / (m_width/2 * m_scaleX) ) );
GLfloat vertex2_theta = -vertex1_theta; // lower right vertex
GLfloat vertex3_theta = vertex1_theta - 180; // lower left vertex
GLfloat vertex4_theta = 180 - vertex1_theta; // upper left vertex

// Now get the rotated vertex angles
vertex1_theta += rotation_angle;
vertex2_theta += rotation_angle;
vertex3_theta += rotation_angle;
vertex4_theta += rotation_angle;

//Calculate the distance from the center (same for each vertex)
GLfloat r = sqrt(pow(m_width/2*m_scaleX, 2) + pow(m_height/2*m_scaleY, 2));

/* Calculate each vertex (I'm not familiar with OpenGL, DEG_TO_RAD
 * might be a constant instead of a macro)
 */
vertexN_x = m_position.x + cos(DEG_TO_RAD(vertexN_theta)) * r;
vertexN_y = m_position.y + sin(DEG_TO_RAD(vertexN_theta)) * r;

// Now you would draw the rectangle, proceeding from vertex1 to vertex4.

Obviously more longwinded than necessary, for the sake of clarity. Of course, duffymo's solution using a transformation matrix is probably more elegant and efficient :)

EDIT: Now my code should actually work. I changed (width / height) to (height / width) and used a constant radius from the center of the rectangle to calculate the vertices. Working Python (turtle) code at http://pastebin.com/f1c76308c