Why does an infrared thermometer display very low temperature when being directed to the outer air?
The reason for this can be seen by examining how an infrared thermometer works. As you mentioned, it measures the infrared radiation, and uses this to determine the temperature. So, with that in mind, we consider what happens in the situations you mention. Namely, you cannot use it to measure the air temperature, because the emissivity of the air is piss-poor. So it will "think" the air is very cold, when it is not. Pointing at the sky, well, there's no IR emission of any great significance, that the device would register.
This can be seen by examining footage taken with a thermal infrared camera. This shows how the world "looks" in the thermal infrared band that these devices use. Using such a camera, one sees that the sky appears black. There is little emission, and there is practically no Rayleigh scattering of sunlight either at these wavelengths (it scatters better in the visible spectrum, esp. toward the blue, which is why the sky is blue).
For pointing at a far-off building, the distance means the amount of IR light reaching the device will be too small for it to make a proper reading.
The sky at night (no clouds) typically measures -48c to -21c at my location (Seattle), currently -44c. The clouds measure -5c to -1c right now. The trees across the road are at 2c.
The reason I can measure the clouds is that the atmosphere is largely transparent to IR, so my thermometer "sees" the radiation from the clouds, but not the atmosphere - the intervening gasses don't tend to absorb or emit photons of IR radiation, so don't interfere with the measurement (the technical term for this is "piss-poor emissivity").
Water is a good emitter - so if it were raining (and, being Seattle, I'm surprised it's not), it'd mainly "see" the radiation coming from the droplets (although the temperature reading might also be wildly inaccurate, because your typical cheapo IR thermometer doesn't actually measure the dominant IR wavelength - which would allow an accurate calculation via the Wien displacement law - but instead measures the total IR emitted at a specific wavelength, extrapolating that back to a temperature via the Stefan-Boltzmann law, which is that the amount of radiation emitted is proportional to the fourth power of the temperature).
As for the -44c of the clear sky: either something up there (eg ice/dust particles in the stratosphere) is emitting at that ambient temperature, or the temperature reading is meaningless. I tend to prefer the former view, as the temperature is remarkably consistent depending on the season :)
If your thermometer went below 0c, I presume you'd see the same.
For completeness, since this came up again:
An infrared thermometer is a thermometer which infers temperature from a portion of the thermal radiation sometimes called blackbody radiation emitted by the object being measured.
To get accurate readings from an infrared thermometer, the emissivity of the body aimed at has to be gauged and applied to it.
By knowing the amount of infrared energy emitted by the object and its emissivity, the object's temperature can often be determined. Infrared thermometers are a subset of devices known as "thermal radiation thermometers".
Emissivity:The ratio of the radiation emitted by the surface at a given temperature to the radiation emitted by a blackbody at the same temperature
Air does not follow a black body radiation curve well and has a very small emissivity, of order of 0.3 for normal humidity and CO2, therefore will be far away from the callibration curve of a given thermometer.
One has to know to what emissivity value the infrared thermometer is calibrated to really have a temperature measurement.
.