OpenGL Math - Projecting Screen space to World space coords

RESOLVED

Here's how to do it exactly, step by step.

  1. Obtain your mouse coordinates within the client area
  2. Get your Projection matrix and View matrix if no Model matrix required.
  3. Multiply Projection * View
  4. Inverse the results of multiplication
  5. Construct a vector4 consisting of

    x = mouseposition.x within a range of window x

    • transform to values between -1 and 1

    y = mouseposition.y within a range of window y

    • transform to values between -1 and 1
    • remember to invert mouseposition.y if needed

    z = the depth value ( this can be obtained with glReadPixel)

    • you can manually go from -1 to 1 ( zNear, zFar )

    w = 1.0

  6. Multiply the vector by inversed matrix created before

  7. Divide result vector by it's w component after matrix multiplication ( perspective division )

        POINT mousePos;
        GetCursorPos(&mousePos);
        ScreenToClient( this->GetWindowHWND(), &mousePos );         
    
        CMatrix4x4 matProjection = m_pCamera->getViewMatrix() *  m_pCamera->getProjectionMatrix() ;
    
        CMatrix4x4 matInverse =  matProjection.inverse();
    
    
        float in[4];
        float winZ = 1.0;
    
    
        in[0]=(2.0f*((float)(mousePos.x-0)/(this->GetResolution().x-0)))-1.0f,
        in[1]=1.0f-(2.0f*((float)(mousePos.y-0)/(this->GetResolution().y-0)));
        in[2]=2.0* winZ -1.0;
        in[3]=1.0;          
    
        CVector4 vIn = CVector4(in[0],in[1],in[2],in[3]);
        pos = vIn * matInverse;
    
        pos.w = 1.0 / pos.w;
    
        pos.x *= pos.w;
        pos.y *= pos.w;
        pos.z *= pos.w;
    
        sprintf(strTitle,"%f %f %f / %f,%f,%f ",m_pCamera->m_vPosition.x,m_pCamera->m_vPosition.y,m_pCamera->m_vPosition.z,pos.x,pos.y,pos.z);
    
        SetWindowText(this->GetWindowHWND(),strTitle);
    

I had to make some adjustments to the answers provided here. But here's the code I ended up with (Note I'm using GLM, that could affect multiplication order). nearResult is the projected point on the near plane and farResult is the projected point on the far plane. I want to perform a ray cast to see what my mouse is hovering over so I convert them to a direction vector which will then originate from my camera's position.

vec3 getRayFromScreenSpace(const vec2 & pos)
{
    mat4 invMat= inverse(m_glData.getPerspective()*m_glData.getView());
    vec4 near = vec4((pos.x - Constants::m_halfScreenWidth) / Constants::m_halfScreenWidth, -1*(pos.y - Constants::m_halfScreenHeight) / Constants::m_halfScreenHeight, -1, 1.0);
    vec4 far = vec4((pos.x - Constants::m_halfScreenWidth) / Constants::m_halfScreenWidth, -1*(pos.y - Constants::m_halfScreenHeight) / Constants::m_halfScreenHeight, 1, 1.0);
    vec4 nearResult = invMat*near;
    vec4 farResult = invMat*far;
    nearResult /= nearResult.w;
    farResult /= farResult.w;
    vec3 dir = vec3(farResult - nearResult );
    return normalize(dir);
}