What are the differences between the OpenGL, GTK and QT libraries?

It's something more fundamental. OpenGL lets you perform low-level drawing (think "draw a line from here to here" rather than "draw a button"). OpenGL operations are also often hardware accelerated and thus very efficient, and the OpenGL API allows you to easily work with 3D.

Choosing between OpenGL and a GUI toolkit like QT or GTK really depends on whether you're trying to draw UI elements for your application or want very fine control over how your graphics are drawn.

Sometimes, GUI toolkits employ OpenGL under the covers in order to draw their on-screen elements.

OpenGL is a library for drawing graphics on a low level. It excels mainly at 3D graphics. The most important competitor to OpenGL is Direct3D (not the whole DirectX).

GL is only really powerful if you have it hardware accelerated. Today, almost every desktop or laptop solution has GL acceleration, but on some systems where the graphics card is integrated into the chipset, it is still rather slow.

For day-by-day hacking, GL can serve many unusual purposes. The reason is that you can leverage the GPU's processing power for graphic related tasks. For anything non-graphical, newer languages like CUDA exist as well to access the GPU.

While GL's real strengths lie in 3D scene rendering, it is often also used for 2D manipulation or composition. The reason is that 2D acceleration architectures of graphics cards often don't reach the power (freedom) or performance of GL. That's why sometimes GL is used for video output (mainly scaling) or blending tasks, even if specific 2D acceleration for these tasks would be available. Mac OS X is a prominent example of a whole desktop (optionally) being rendered through GL to allow appealing effects without high CPU load. But it can also be used for example to blend anti-aliased text glyphs onto a textured background.

As soon as you do a graphical, performance critical task, you will probably want the GPU to help out (possible even if you don't draw on screen) and then OpenGL is the clean, standardized, cross-platform answer to your needs. Today's GUI toolkits mostly don't offer you GPU-accelerated drawing, but the inclusion of GL rendering into their widgets. So you can use them for UI drawing and include your performance critical realtime graphics where it suits. Building a whole UI inside GL is a pain, but there are also solutions for that available (typical use cases are games with in-game UI).