Why is there a voltage drop across things when no current is flowing?

I guess more generally I'm confused as to why things with zero current going through them have a voltage drop at all as V=IR.

Ohm's law applies to ohmic devices; if the voltage across a device is proportional to the current through, the device is ohmic otherwise it isn't.

Ohm's law is not a universal law. For example, Ohm's law does not apply to capacitors, inductors, diodes, transistors, vacuum tubes, etc. etc.

An open (ideal) switch is not an ohmic device since the current through $(0\mathrm{A})$ is not proportional to the voltage across. However, one can think of an open switch as the limit as $R \rightarrow \infty$ of a resistor.

A closed (ideal) switch is not an ohmic device since the voltage across $(0\mathrm{V})$ is not proportional to the current through. However, one can think of a closed switch as the limit as $R \rightarrow 0$ of a resistor.

Using the water analogy of circuits, this is like asking how two sections of connected pipe could be at different heights when there is no water flowing between them.

This is just a conceptual mistake. The Ohm's law states that the voltage drop developed across a conductor is proportional to the current flowing through the conductor, the proportionality being a property of the conductor itself- it's resistance. This is what we write mathematically as

$$V=IR$$

Now, when you connect an open switch and a resistor to the battery, the open switch allows infinite resistance to be there in the circuit. Hence the current flowing through the circuit will be

$$I=\frac{V}{R}=\frac{V}{\infty}=0$$

This means no current flows through the conductor. But the applied voltage has to be there somewhere. It has to be so. So the entire voltage appears across the resistor.

Now, what happens if you apply voltage across a conductor with zero resistance? Then the voltage will be

$$V=IR=I_\textrm{max}(0)=0$$

where $I_\textrm{max}$ is the maximum current that flows through the circuit.

These arguments can be visualized so easily as follows. Imagine the circuit. The switch is open. This means, the charges could not flow through the circuit and complete the circuit. This is due to the gap in between the switch. So the applied energy is not converted to current even though the resistance is still there. Hence voltage will be there, where you applied it.

Now, in the second case, there is no resistance. This means the supplied energy (the voltage) is completely utilized by the charges to flow through the circuit. There is nothing left so as to be detected by any external circuit (a voltmeter probably). Hence one cannot see any voltage difference between the conductor. You can see energy somewhere if it is there. But if it's being utilized, then you will not see it there, but its there in some other form- here it is the kinetic energy of the charges.

All we invoked here is the energy conservation (not precisely...).