Do batteries lose voltage as they're used up?

Both effects occur as a battery is drained. The open circuit voltage goes down and the internal resistance goes up. Note that open circuit voltage is specifically measuring just the voltage the battery puts out with the internal resistance taken out of the equation. That is because there is no current thru that resistance, hence no voltage drop across it. Any decent voltmeter will have at least 10 MΩ input resistance, which is so way more than even a dead battery as to not matter.

All that said, different battery chemistries have different characteristics regarding both these parameters as they are drained. NiCd and NiMH have rather flat discharge curves after a short initial period. That means the open circuit voltage doesn't drop much for most of the discharge cycle even as the stored energy is getting steadily lower. These batteries then show a rather steep falloff in voltage as the last 10% or so of energy is drained. For a NiMH or NiCd therefore, it's tricky to determine a state of charge just from the voltage.

Other chemistries have a more linear discharge curve (voltage as a function of accumulated Coulombs drained at a fixed current). Old fashioned carbon-zinc cells are more like this. Usually, there is a significant temperature dependence too, both in terms of voltage and capacity.

Yes, batteries can get complicated.

Your 9V battery will indeed give a lower voltage reading when it's exhausted and that's not just because of higher internal resistance; you may read 6 or 7V even with a very high impedance DMM. I'm not sure you can go as low as 1.5V; the increased internal resistance makes that in the end you can hardly draw any energy from it anymore, so I expect that the voltage will go asymptotically to a somewhat higher voltage. Even so, a 9V depleted till 1.5V will never be able to supply the current a 1.5V battery can supply.

Actually, resistance dramatically changes as the battery is used up. The voltage will go down with use, but in many applications the increased internal resistance will render the battery unusable long before the reduced voltage does.