Does a battery powered resistor have Johnson Noise?
yes. Even a resistor with no power source will generate Johnson noise. This is thermal noise due to random movement of electrons in the resistor itself.
f in the formula is the bandwidth across which you wish to calculate the noise, it has nothing to do with the signal applied (or not) to the resistor.
The bigger the resistance of a resistor or the wider the bandwidth of the sample measurement, the higher the RMS thermal noise of the sample measurement. Measuring samples for a longer time duration, effectively increases the sample measurement bandwidth and also increases the noise in the measured sample. Measuring for less time gives less sample bandwidth and thus less RMS noise in the sample. This effect is due to the act of sampling the noise. If you could take a measurement that ran for the entire duration of the universe, then the measurement would capture everything -- but you'd only get the measurement result at the end of the universe. Taking a smaller sample gives a less accurate result because it omits detail, but the measurement result is achieved faster. The bandwidth (or delta-f) term in Johnson's equation reflects the effect of how long an uncorrelated noise source is measured.
If you put an oscilloscope probe on a noise source and estimate its noise based on the measured vertical deflection, there will be more RMS and more peak-to-peak noise in the sample if the horizontal sweep rate is slower. If you check the datasheet of a reference such as the Maxim Integrated MAX6126, on page 13 there are typical operating characteristics graphs showing exactly that measurement, "Output Noise (0.1Hz TO 10Hz)". (I am an applications engineer at Maxim Integrated.) Whenever we provide noise measurement characteristic graphs like this, we have to provide the measurement bandwidth as one of the test conditions because it will change with different test conditions.
An oscilloscope has limited bandwidth to what range of frequencies it can capture for a given sweep. For a horizontal sweep rate of 1 second per division, and a display width of 10 horizontal divisions, the lowest measurable frequency would be 0.1Hz (all 10 divisions). The highest measurable frequency is deemed to be 10Hz (1/10th of 1 division) based on the limit of the display's resolution. (At higher frequencies there would also be some additional bandwidth limits in the analog / data acquisition path, but they're negligible at 10Hz.)
For many types of precision analog and data acquisition systems (ADCs, DACs, precision op amps) there are noise specifications given in units of nV/sqrt(Hz) for Johnson noise and nA/sqrt(Hz) for shot noise. Both of these types of noise vary with the measurement bandwidth.
Johnson-Nyquist noise has nothing to do with the power supply. The \$f\$ in the formula is the bandwidth of the noise, not any applied frequency.