How to implement a ground fog GLSL shader

This is a pretty standard application of atmospheric scattering.

It is discussed under the umbrella of volumetric lighting usually, which involves transmittance of light through different media (e.g. smoke, air, water). In cutting-edge shader-based graphics, this can be achieved in real-time using ray-marching or if there is only one uniform participating medium (as it is in this case -- the fog only applies to air), simplified to integration over some distance.

Ordinarily you would ray-march through participating media in order to determine the properties of light transfer, but this application is simplified to assume a medium that has well-defined distribution characteristics and that is where the coefficients you are confused about come from. The density of fog varies exponentially with distance, and this is what b is controlling, likewise it also varies with altitude (not shown in the equation directly below).

   
_{(source: iquilezles.org)}

What this article introduces to the discussion, however, are poorly named coefficients a and b. These control in-scattering and extinction. The author repeatedly refers to the extinction coefficient as extintion, which really makes no sense to me - hopefully this is just because English was not the author's native language. Extinction can be thought of as how quickly light is absorbed, and it describes the opacity of a medium. If you want a more theoretical basis for all of this, have a look at the following paper.

With this in mind, take another look at the code from your article:

vec3 applyFog( in vec3  rgb,      // original color of the pixel
               in float distance, // camera to point distance
               in vec3  rayOri,   // camera position
               in vec3  rayDir )  // camera to point vector
{
    float fogAmount = c*exp(-rayOri.y*b)*(1.0-exp(-distance*rayDir.y*b))/rayDir.y;
    vec3  fogColor  = vec3(0.5,0.6,0.7);
    return mix( rgb, fogColor, fogAmount );
}

You can see that c in this code actually a from the original equation.

More importantly, there is an additional expression here:

   

This additional expression controls the density with respect to altitude. Judging by your implementation of the shader, you have not correctly implemented the second expression. camFrag.z is very likely not altitude, but rather depth. Furthermore, I do not understand why you are multiplying it by the b coefficient.

I found a method that gives the result I was looking for.

The method is described in this article of Eric Lengyel : [link]http://www.terathon.com/lengyel/Lengyel-UnifiedFog.pdf

It explains how to create a fog layer with density and altitude parameters. You can fly through it, it progressively blends all the geometry above the fog.