Alkaline battery noise varies with temp

Considering that noise variation with temperature is a fundamental property of matter, all the things (which includes alkaline batteries) will have noise proportional to temperature. All resistances have thermal noise, and all batteries have resistance, and their noise is more or less from that internal resistance. The voltage noise of a battery (or resistor) is: $$ V_{noise}=\sqrt{\frac{4hfR\Delta v}{e^{\frac{hf}{kT}} -1}} $$

where h is planck's constant, f is the frequency, R is the internal resistance of the cells or cell, ∆v is the bandwidth, k is Boltzmann's constant, and T is temperature in kelvin. As you can see, lowering the temperature lowers the noise. This is true for everything, there is nothing happening here unique to batteries. This noise is called Johnson-Nyquist noise.

As for which chemistry has the lowest noise, there is no meaningful difference in theory. In practice, Nickel-Cadmium cells have the lowest voltage noise. However, this is purely due to that chemistry also having the lowest internal resistance. As you can see from the earlier equation, lowering resistance will lower noise over all. Alkaline cells have relatively high internal resistance, so it is not surprising they would be noisier as a chemistry. Note that this means cell size is as important to the voltage noise as cell chemistry. Larger cells have lower internal resistance and therefore lower noise.

But don't take my word for it. Take NIST's. They did a study on the noise of batteries, and there are nice graphs for those curious in that paper, but after substantial measurements all the way down to the thermodynamically limited noise-floor, they concluded that battery voltage noise is essentially in agreement with the expected Johnson-Nyquist thermal noise one would expect from the cell's internal resistance.

Edit: Whoops, I forgot that the entire question was about the noise increasing once it got cold enough. A battery's internal resistance increases as it gets cold, and decreases as it gets warm. This mechanism is chemical in nature, and likely could vary between different constructions of the same chemistry. In general, temperature can increase the internal resistance a lot once you get cold enough. The internal resistance is ultimately determined by the rate the chemical reaction can occur, and the colder the battery, the slower the reaction. It's a safe bet to look at a cell or chemistry's internal resistance vs. temperature, this should give you a good idea of how warm you need to keep the cell. There is going to be a 'sweet spot' where the noise is lowest. Warmer and the temperature increases the noise more than the internal resistance decreases, colder and the internal resistance increases more than the noise decreases.

EDIT2: It looks like an alkaline cell's internal resistance doubles (or at least an AA cell) going from 20 degrees C to 10. This is far too small to account for the several orders of magnitude noise increase.

Sorry. Something weird is going on. Thermocouple effects perhaps?