How to recalculate axis-aligned bounding box after translate/rotate?

Yep, you can transform the eight corner vertices and do min/max on the results, but there is a faster way, as described by Jim Arvo from his chapter in Graphics Gems (1990).

Performance-wise, Arvo's method is roughly equivalent to two transforms instead of eight and basically goes as follows (this transforms box A into box B)

// Split the transform into a translation vector (T) and a 3x3 rotation (M).

B = zero-volume AABB at T
for each element (i,j) of M:
   a = M[i][j] * A.min[j]
   b = M[i][j] * A.max[j]
   B.min[i] += a < b ? a : b
   B.max[i] += a < b ? b : a
return B

One variation of Arvo's method uses center / extent representation rather than mix / max, which is described by Christer Ericson in Real-Time Collision Detection (photo).

Complete C code for Graphics Gems article can be found here.

To do that you have to loop over every vertex, calculate its position in the world (multiply by modelview) and find the minimum / maximum vertex coordinates within every object (just like when you compute it for the first time).

You can scale your AABB a bit, so that you don't have to recalculate it - it is enough to enlarge it by factor sqrt(2) - your rotated object then always fits in AABB.

There is also a question in which direction you rotate(?). If always in one then you can enlarge AABB only in that direction.

Optionally, you can use bounding spheres instead of AABBs. Then you don't care about rotation and scaling is not a problem.

Simply recompute the AABB of the transformed AABB. This means transforming 8 vertices (8 vertex - matrix multiplications) and 8 vertex-vertex comparisons.

So at initialisation, you compute your AABB in model space: for each x,y,z of each vertex of the model, you check against xmin, xmax, ymin, ymax, etc.

For each frame, you generate a new transformation matrix. In OpenGL this is done with glLoadIdentity followed by glTransform/Rotate/Scale (if using the old API). This is the Model Matrix, as lmmilewski said.

You compute this transformation matrix a second time (outside OpenGL, for instance using glm). You also can get OpenGL's resulting matrix using glGet.

You multiply each of your AABB's eight vertices by this matrix. Use glm for matrix-vector multiplication. You'll get your transformed AABB (in world space). It it most probably rotated (not axis-aligned anymore).

Now your algorithm probably only work with axis-aligned stuff, hence your question. So now you approximate the new bounding box of the transformed model by takinf the bounding box of the transformed bounding box:

For each x,y,z of each vertex of the new AABB, you check against xmin, xmax, ymin, ymax, etc. This gives you an world-space AABB that you can use in your clipping algorithm.

This is not optimal (AABB-wise). You'll get lots of empty space, but performance-wise, it's much much better that recomputing the AABB of the whole mesh.

---

As for the transformation matrix, in drawObjectPlayer:

gLLoadIdentity();
glTranslatef(objPlayer.position.x, objPlayer.position.y, objPlayer.position.z);
glRotatef(objPlayer.rotation.y + 180.0f, 0.0f, 1.0f, 0.0f);
glGetFloatv(GL_MODELVIEW_MATRIX, mvMatrix);
// Now you've got your OWN Model Matrix (don't trust the GL_MODELVIEW_MATRIX flag : this is a workaround, and I know what I'm doing ^^ )

gLLoadIdentity(); // Reset the matrix so that you won't make the transformations twice
gluLookAt( whatever you wrote here earlier )
glTranslatef(objPlayer.position.x, objPlayer.position.y, objPlayer.position.z);
glRotatef(objPlayer.rotation.y + 180.0f, 0.0f, 1.0f, 0.0f);
// Now OpenGL is happy, he's got his MODELVIEW matrix correct ( gluLookAt is the VIEW part; Translate/Rotate is the MODEL part
glCallList(gameDisplayLists.player); // Transformed correcty

I can't explain it further than that... as said in the comments, you had to do it twice. You wouldn't have these problems and ugly workarounds in OpenGL 3, btw, because you'd be fully responsible of your own matrices. Equivalent in OpenGL 2:

glm::mat4 ViewMatrix = glm::LookAt(...);
glm::mat4 ModelMatrix = glm::rotate() * glm::translate(...);
// Use ModelMatrix for whatever you want
glm::mat4 ModelViewMatrix = ViewMatrix * ModelMatrix;
glLoadMatrix4fv( &ModelViewMatrix[0][0] ); // In OpenGL 3 you would use an uniform instead

Much cleaner, right?