How can I tell if OpenGL is installed on a windows machine?

I believe Windows comes with OpenGL drivers. http://www.opengl.org/wiki/Getting_started seems to confirm this:

If you are running Windows 98/Me/NT/2000/XP/2003/Vista, the OpenGL library has already been installed on your system.

The above page shows the default location for opengl32.dll on the different versions of windows too. E.g. windows\system32\opengl32.dll So you could look there for them on each machine.

If you want to determine the version on each system then I think you'll need a utility.

Found OpenGL Extensions Viewer. But it's a utility answer.

There is another utility called GPU Caps Viewer on Geeks3D.

GPU Caps Viewer is a graphics card information utility focused on the OpenGL, Vulkan, OpenCL and CUDA API level support of the main (primary) graphics card. For Vulkan, OpenCL and CUDA, GPU Caps Viewer details the API support of each capable device available in the system.