How to properly handle refraction in raytracing
EDIT: I've figured that the previous version of this was not entirely correct so I edit the answer.
After reading all the comments, the new versions of the question and doing some experimentation myself I produced the following version of refract
routine:
float3 refract(float3 i, float3 n, float eta)
{
eta = 2.0f - eta;
float cosi = dot(n, i);
float3 o = (i * eta - n * (-cosi + eta * cosi));
return o;
}
This time calling it does not require any additional operations:
float3 refr = refract(rayDirection, normal, refrIdx);
The only thing I am still not sure is the inverting of the refractive index when doing the inside ray intersection. In my test the produced image haven't differ much no matter I inverted the index or not.
Below some images with different indices:
For more images see the link, because the site do not allow me to put more of them here.
I am answering this as a physicist rather than a programmer as haven't had time to read all the code so won't be giving the code to do the fix just the general idea.
From what you have said above the black ring is for when n_object is less than n_air. This is only usually true if you are inside an object say if you were inside water or the like but materials have been constructed with weird properties like that and it should be supported.
In this type of situation there are rays of light that can't be diffracted as the diffraction formula put the refracted ray on the SAME side of the interface between the materials, which obviously doesn't make sense as diffraction. In this situation the surface will instead act like it's a reflective surface. This is the situation that is often referred to as total internal reflection.
If being fully exact then almost ever refractive object will also partially reflective too and the fraction of light that is reflected or transmitted (and therefore refracted) is given by the Fresnel equations. For this case though it would still be a good approximation to just treat is as reflective if the angle is too far and transmitting (and therefore refractive) otherwise.
Also there are situations where this black ring effect can be seen if reflection is not possible (due to it being dark in those directions) but light that is transmitted being possible. This could be done by say taking a tube of card that fits tightly to the edge of the object and is pointed directly away and only shining light inside the tube not outside.