Why is infrared radiation associated with heat?

It sounds like you know some of the most important summary points about blackbody radiation, but here is a reference on the subject, since I will be talking almost entirely about blackbody radiation: https://en.wikipedia.org/wiki/Black-body_radiation

Given any temperature, there is a certain emission spectrum (see https://en.wikipedia.org/wiki/Planck%27s_law) describing the mixture of photons which a body at that temperature emits. As you already noted, the peak frequency of this spectrum increases as the temperature increases. It is also of interest to note that an increase in temperature increases photon emission at ALL frequencies, not just higher frequencies.

The short form answer to your question is that we live in on Earth, where temperatures tend to be in the 200-400K range. Even a campfire doesn't make it much above 1500K. At all of these temperatures (yes, including the campfire) the VAST majority of the energy radiated is in the infrared range. If you put a filter between yourself and a campfire which absorbed all visible light and transmitted all infrared light, you would feel just as warm. So it is natural for us to associate infrared radiation with heat. The sun is the only everyday example of something that warms us noticeably with visible light, and sunlight already holds its own unique place in the human experience. Sunlight feels warm. A physicist can get out sensitive instruments and observe that in fact all light warms us slightly, but as far as what we can feel with our own nerves goes, it is only infrared and sunlight that seem warm.

If we were plasma beings that inhabited the core of the sun, perhaps we would associate visible light (or some energetic subatomic particle or other...) with the transmission of thermal energy, but we aren't and we don't.

In the end, all light transmits energy, and heat is just energy in the form of atoms exercising their degrees of freedom. So there is no clear cut distinction between the way infrared radiation interacts with heat and the way any other radiation does. But over most of the wide range of commonly studied environments, heat is mostly ratiaded as infrared photons. Thus the association.

Regarding your thought about incandescent lights vs more efficient alternatives, we replaced our 60-100 Watt bulbs with 7-20 Watt bulbs, so they really don't warm us up anywhere near as much as the old ones. If we had replaced the bulbs with equivalent wattages, then your thought would be correct, but we would be blinded by our lamps!

You will want to refer to the blackbody curve, which will have a broad spectrum and a peak wavelength. Consider the sun for a moment. At a temperature of around 6000 Kelvin the peak wavelength of its blackbody curve will appear around 500nm, which is blueish-green light. But that is obviously not the only wavelengths that the sun is emitting. The blackbody curve has a long tail far out into the infrared and beyond (the sun also emits radio waves). The hotter the blackbody, the more the peak of the curve shifts toward shorter wavelengths. But this is very important and something I did not appreciate before either: the amount of infrared radiation also increases with hotter temperatures. So just because the peak has shifted into the visible does not mean that most of the light being emitted is visible. Most of the light being emitted by a blackbody is always infrared light.