Mathematical joke regarding three people leaving a house

Note that there are $2$ people who enter the house at first. Then $3$ people leave the house.

The joke probably lies in the fact that the mathematician assumes that there were initially no people inside the house (The others do too), and so then there are: $$2-3=-1\text{ people}$$ In the house at the moment. Therefore, if one person enters the house, the amount of people in the house is: $$-1+1=0 \text{ people}$$ Thus, the house will be empty.

I think the point of the punchline is that the mathematician simply solves the math problem as observed, and appears totally unconcerned with the impossibility of having -1 people in a house in the real world.

Sciences like physics and biology are about explaining the real world, but in mathematics explaining the real world is not a requirement.

From my understanding, it's just a take on stereotypes of physicists, biologists, and mathematicians. Physicists are generally concerned with real-world measurements, and may often attribute issues to experimental error (counting the wrong number of people here). Biologists end up attributing the extra person due to sex (reproduction). The mathematician sees $2-3=-1$ people in the house, so if one more person enters, there will be $-1+1=0$ people.