Should I always use GL_CULL_FACE?

Enabling face Culling (usually back-face) has a significant impact on the rendering performance.

Now the question to be answered is then "why isn't culling enabled by default or why isn't used always?" there are several factors which determine whether enabling culling is feasible or not. One is the model being rendered and how it is viewed.

For every polygon in the model, if only one face either front or back is visible (when the model is viewed from all possible angles) then we can cull the other face.

For Ex: "Cube", no matter if you are looking from inside or outside of it, when rotated, you can see only one side of every 6 polygons which make-up the cube.

Another Ex: "Sails of a ship Model" as in my Sea-Wars(3d) game, the sails of a ship model is a surface built with several flat polygons. When a boat moves around in the water, one can see both sides of the sail. In this case culling either of the faces is not desirable.

Hence culling can be enabled for all the models in the scene which follow cubes semantics and disabling for all other models.

P.S. Face Culling should be used whenever possible.

If all your models are defined correctly (convex, with consistent vertex winding), culling will never hurt performance, and will nearly always help performance.

Imagine you are rendering a cube, with culling disabled. The faces on the back (far side) of the cube get rendered facing away from you, and whatever back face material attributes you've chosen get applied. Then the faces on the front of the cube get rendered: since the back faces that were just rendered are the inside of the cube, they are 100% occluded by the front faces.

If the cube was oriented differently, the front faces might be rasterized first, but the back faces will still be processed and then the fragments rejected by the z buffer.

Net result: you rasterized 2x as many fragments as necessary. This gets to be a big deal with large numbers of polygons or complex pixel shaders. Face culling allows you to completely avoid rasterizing faces that you know won't show up.

If you have a character model with ~100,000 polygons, face culling halves the amount of work the pixel hardware has to do every frame. This is a big savings if you have big surface shaders.

If you're working with small models and no shaders, culling really doesn't matter these days. But it's good practice to enable anyway.