What are colors?

This has as much to do with biology as with physics. The long answer on the biology is here. The biology in summary: the human eye has only three different "color-sensitive" elements, and uses a complex combination of the amount of response it sees from each of these to assign a "color" to the image.

Because there are only three sensitivities in the human eye, there are a variety of different techniques using only three basic (in some schemes called primary) colors to represent colors to humans.

The actual frequency of the light emitted from the part of the rainbow we call green may have the same effect on the human eye as something we get by mixing our blue crayon with our yellow crayon on a piece of paper. But it is easy to build a detector which will trivially differentiate between monochromatic green from one slice of a rainbow, and a mixture of the light reflected from blue and yellow crayon pigments.

In principle, humans might have evolved a different eye with four or five different "color" detectors, in which case the schemes needed to make color images would probably need to have four or five basic colors, and the images we see from our current three color representations would seem to be washed out, missing something important. But the eye didn't develop that way.

RGB is a property of our eyes, not of physics. Our eyes only have 3 different types of color-sensing cells, so we can only perceive 3 regions of the rainbow, and mixtures of those regions. Yellow light falls between red and green, for instance, and triggers both types of cells, but we don't have cells that respond only to yellow.

Mantis shrimp can perceive 12 different colors, as well as the polarization of light, which helps them see camouflaged or transparent prey. The military does the same thing with hyperspectral cameras: http://www.bbc.co.uk/news/10175960