Does battery voltage actually get lower when connected to a load, or does it just appear to do so?

Yes, it does get lower.

The effect you see is called internal resistance:

A practical electrical power source which is a linear electric circuit may <...> be represented as an ideal voltage source in series with an impedance. This impedance is termed the internal resistance of the source.

Put simply, a battery is not an ideal voltage source. A typical battery (i. e. non-ideal voltage source) will look like this:

What you are measuring is voltage between terminals A and B. According to Ohm's Law:

$$ U_{AB} = E * \frac{R}{R+r} $$

• When there is no circuit, you can imagine your voltmeter's internal series resistance \$ R_{volt} \$ taking the role of \$ R \$. However, \$ R_{volt} \$ is usually so large (tens or hundreds of megaohms) compared to \$ r \$ (usually fractions of an ohm) that \$ \frac{R_{volt}}{R_{volt}+r} \$ tends to 1, hence measured open-circuit voltage tends to the battery's internal (true) voltage \$ E \$.

• When there is a closed circuit with equivalent series resistance of \$ R \$, you will be able to see that the measured voltage \$ U_{AB} \$ drops proportionally to \$ R \$, in accordance with the above formula.

So, the voltage drop is real — the measured voltage is what your load gets. The more current it draws from the battery, the lower is voltage it gets.

When the battery is open you are measuring an open cell voltage. When the battery is in the system it's closed cell voltage under load. You are dropping some voltage across the internal impedance of the battery because your system is drawing current when the measurement is being made (so at the terminals the voltage is indeed lower). So both measurements MCU and multimeter are correct, the difference is that the multimeter is >1Mohm load while the MCU is much lower (since probably drawing at least mAs of power).

There may be another effect at play. Batteries do exhibit a recovery phenomenon where if left open cell with no load some of the voltage will recover after a time interval.

Every battery has a certain amount of output resistance. What happens if current flows through a resistor? Yes, a voltage drop! So the more current you draw from the battery, the lower the output voltage is.