Why is the voltage measurement of this circuit different when the switch is on?
First of all, well done for doing the practical experiment. It's good to see people trying things and wanting to discover the answers to their observations.
It may be easier to visualise this with a circuit. As it has been pointed out in a previous answer, batteries have internal resistance. So if you add a load on to it, you create a voltage divider. As the battery becomes more and more drained, the internal resistance increases, which stops it being able to power the load.
simulate this circuit – Schematic created using CircuitLab
Look at the above schematic. We all know Ohm's Law, which is V = I*R. If we give values to the resistors we can calculate how much the drop will be when you close the switch. If we say the battery internal resistance is 0.5 Ohms, and the load is 100 Ohms we can find out the current in the circuit. To do this, we re-arrange the Ohm's Law equation for I: I = V/R = 9/(100+0.5) = 0.0896 A, or 89.6mA. Using Ohm's Law again (you will find that this is probably the most useful equation you will come across in electronics!), we can find the voltage drop over the battery resistance:
Remember: V = I*R = 0.0896*0.5 = 0.0448V. Take this away from the 9V the battery first supplied and you get the voltage you will be measuring when the switch is closed: 9-0.0448 = 8.95V.
Using this knowledge, if you wanted to further your experiments, why not get a known resistor as a load, and power it using different batteries. With your multimeter, you can measure the current and the voltage, which will give you all the numbers you need to calculate the internal resistances of your batteries.
As Marcus Muller points out, temperature can have an effect on this too, so why not do a before and after test, calculate your internal resistance before and after putting them in a fridge/freezer, and see how much it changes. Try it on a few different types of batteries too... There are a lot of cool experiments you can do which will increase your understanding of what is going on, and improve your skills at solving circuits.
Keep it up and good luck!
This is an extremely nice experiment!
I have an idea to add to jonk's comment:
It is super-good that you are testing out what happens like this! As already provided, the answer is that the battery is under some strain when powering the lights and it's voltage "droops" a little under that load. When a battery gets nearer its end-of-life, it will droop even more.
Put the batteries in the fridge for an hour or so, until they're cold (you can freeze them, too, but don't go lower than -20 °C), and repeat your experiment. You'll see that they droop a lot harder now! Turn off and wait until they have room temperature again: They should start working just like before.
What's happening?
In a battery, a chemical reaction leads to the two battery contacts having different electrical potentials – there's a voltage between them!
When you connect two things, your battery contacts, that are at different voltages, current starts flowing. That's what lights up your light bulb!
Now, imagine there's a small resistor (like, 2Ω) between your battery and the positive probe of your multimeter. It's not actually there, but you "feel" its existence:
A resistor shows a voltage drop when a current flows through it. In your case, a couple hundred milliampere flow through the bulb, the switch and back into your battery – and that leads to a couple hundred millivolts in voltage drop over that "imagined" resistor.
That's what we call "internal resistance". It's the imperfection of a voltage source (like your batteries) that leads to lower voltages the more current you draw.
Internal resistance can be many things – first of all, real batteries are made of real materials and real materials have resistance. But for batteries, that's usually just a small part of the internal resistance. The larger part is that to make a current flow, the chemical reactions (and ion wandering) inside must happen fast enough. If there's more current draw than the chemical reaction can sustain, the voltage drops.
Now, when you cooled down your batteries, you slowed down all the chemical reactions inside, and especially how fast charged atoms can wander inside the battery. That's the reason why we have fridges and freezers: With all chemical reactions being slowed by lower temperature, foodstuff doesn't go bad as quickly, because all the things that make food go bad (that is, bacterial growth, and chemical decomposition of things) simply happen in slow motion.
With the chemical reaction in the battery slowed down, the battery simply can't "keep up" as good with the current draw, and the voltage drops even further than with a warm battery.
Batteries are not ideal. They have internal resistance. When current flows through it, voltage drops.