What gives things color?
Every Element/atom has a different electron configuration. This gives the valence electrons unique energy levels and arrangements. When electrons absorb energy they are excited to certain and again unique energy levels. When The energy is released it gives a photon a certain frequency which we perceive as a certain color.
A lot of things can affect the color of an object. As you've mentioned, absorption plays an important role in determining what part of the visible spectrum gets subtracted from the color your eyes perceive. Optical bandgap arising from the microstructure of materials determines what portion of the spectrum is absorbed. It is closely related to the electronic bandgap, electronic density of states, and E-k dispersion relation of crystalline materials. In addition, crystallinity and size of crystallites can also affect a material's electronic characteristics. For example, crystalline silicon has an indirect bandgap of ~1.1eV, whereas amorphous silicon has a direct bandgap of >1.5eV but with many defect states within the gap. The larger bandgap of amorphous silicon means thin film amorphous silicon can appear to be pink-ish when compared to its crystalline counterpart.
On the other hand, emission of light via black body radiation, and photoluminescence can also change the appearance of an object.
For black body radiation, think of the sun. A hotter object appears to be brighter, and whiter compared to a colder object. That's because the peak of radiation spectrum shifts to higher energies when a piece of material is heated. Then, the appearance of an object depends also on external conditions.
Photoluminescence is the ability a material has to emit photons at the same or different energies after absorbing light. Take a look at your highlighter pens, and you'll get an idea of how the ink in a highlighter pen absorbs light at higher energies, and emits light at a relatively fixed energy. For example, a yellow highlighter pen has ink that emits light that appears yellow.
Things don't really have "colour". Colour is a perception that we have of light entering our eyes.
We use our eyes to perceive the world, and in our eyes are a series of receptors (photoreceptors). These are the rod, and cone-shaped cells which exist in your retina. These cells send electrical signals to our brain, via the optic nerve, and it is a combination of these three components which do the perceiving. This model can be used to explain why people who have differing numbers of these photoreceptors can perceive the world differently.
However to answer the question as you meant it you should consider light as just another part of the electromagnetic spectrum. We developed to sense things in the region of the e/m spectrum which was most abundant to us. The largest source of radiation throughout our evolution has been the sun. The Suns peak energy output is in the visible light range, most living things have adapted mechanisms to use this radiation in a positive fashion, Animals through our eyes, and plants through photosynthesis.
Conversely, this region of the electromagnetic spectrum which has shaped the way we have developed is also very useful as it interacts with matter in some interesting ways. The first way is the excitation and relaxation of matter emits radiation in a close band to the visible spectrum. This gives rise to light sources such as halogen, and inert gas lights as well as lasers. Secondly, light also interacts with electrons which are free to move along molecular sized pathways in the movement of molecules (bending and stretching of bonds).