Black and white matters. But why and how?

I know black conducts heat while white reflects it.

The correct term is "black absorbs light while white reflects it".

We have named colors of light we see in the visible spectrum .

White reflects most of the energy falling from the visible spectrum, black absorbs it. When the energy of light is absorbed it turns into heat . Any material painted black will absorb this heat further and its temperature will be raised but it will depend on the material how far the heat is transferred. If it is metal painted black, metal is a good conductor of heat and will distribute the energy fast on the whole body.

But they are colors after all.

They change the surface properties of materials on which they are painted thus changing the ability of absorption and emission of radiation.

The energy coming from the sun covers a much larger electromagnetic spectrum than the visible. The visible has about half of the energy coming from the sun on the surface, as seen in the link.

So a metal door in the sun will transfer the heat of the visible spectrum to the interior if painted black, will reflect it back and keep the interior cooler if painted white. It is a good reason for painting roofs and walls white in hot countries. A white car is also better in hot countries for this reason .

It is not always sure that the color properties ( absorption/reflection) are followed by the invisible part of the sun spectrum, infrared or ultraviolet. Each paint has to be studied as far as its response to the impinging radiation to be used efficiently for thermal protection.

You have it backwards. You're coming from the point of view that being black makes something good at absorbing radiation and being white makes it bad and asking why this should be so. It's exactly the other way around: being good at absorbing radiation (in particular, visible light) makes something black; being bad at absorbing radiation (i.e., good at reflecting it) makes something white.

Now, you can ask what makes something good or bad at absorbing radiation but that's a whole 'nother question.

But, as far as I know, colors don't have any special "substance" in them, which might trigger the sudden absorption of heat or reflection of the same.

This statement appears to be the crux of your confusion, and it is false.

When you paint an object black, or white, or any other color, you coat that object with a thin layer of a substance which, in fact, absorbs particular wavelengths of electromagnetic radiation, and reflects others. The color you see is a consequence of the coating doing this, and which color is dependent entirely on the chemical composition of the paint.

Now, heat is not electromagnetic radiation. Heat is random molecular motion, which can both cause, and be caused by, EM radiation. All wavelengths of EM radiation can transfer heat, but some are better at it than others. Infrared light is sometimes called heat radiation because it is particularly good at transferring heat within our familiar Earth-surface environment, because many substances found in this environment strongly absorb it. In particular, paint which is black (== paint which strongly absorbs all EM wavelengths corresponding to "visible light") will probably also absorb infrared light.

And finally, heat doesn't have to be transferred by electromagnetic radiation. Heat can also transfer by conduction between two objects in direct contact: molecules of the hotter object randomly bump into molecules of the colder object and transfer some of their energy.¹ This process is not dependent on what wavelengths of EM radiation are absorbed by the colder object. This is the primary way paint heats up a painted object, and this is why the underlying color of the object doesn't matter.

¹ technically this process does also involve exchange of "virtual" photons, but this is a detail which is usually ignored for macroscopic objects.