How to check if an object lies outside the clipping volume in OpenGL?

Apply the model-view-projection matrix to the object, then check if it lies outside the clip coordinate frustum, which is defined by the planes:

    -w < x < w
    -w < y < w
     0 < z < w

So if you have a point p which is a vec3, and a model-view-projection matrix, M, then in GLSL it would look like this:

    bool in_frustum(mat4 M, vec3 p) {
        vec4 Pclip = M * vec4(p, 1.);
        return abs(Pclip.x) < Pclip.w && 
               abs(Pclip.y) < Pclip.w && 
               0 < Pclip.z && 
               Pclip.z < Pclip.w;
    }

To determine if a given point will be visible on the screen, you test it against the viewing frustum. See this frustum culling tutorial:

http://www.lighthouse3d.com/tutorials/view-frustum-culling/

For anyone relying on the accepted answer, it is incorrect (at least in current implementations). OpenGL clips in the z plane the same as the x and y as -w < z < w (https://www.khronos.org/opengl/wiki/Vertex_Post-Processing).

The two tests for z should then be: std::abs(Pclip.z) < Pclip.w

Checking for zero will exclude all the drawn points that are closer to the near field clip plane than the far field clip plane.