Why is it that when a chalk board gets cleaned, the area that used to have chalk is the cleanest?

Here's my opinion:

If you notice, the same phenomenon happens with a marker and a whiteboard. So what I think the reason is, that due to adhesion forces among the chalk particles(or for that matter, between the particles/layers of the liquid ink), they tend to remain attached to each other and don't let go easily. So, when your duster removes the upper layers of the chalk particles, these particles exert an upward pulling force to the layer beneath them(The forces I've talked about a few lines above), because of which more chalk particles are removed from that area.

I have an argument to support my answer: If you use a cheap and rough chalk, this phenomenon doesn't take place because there are not enough upward forces in the chalk particles due to the powdery and rough nature of the chalk.

My guess is that the chalk you have just applied binds to the existing chalk particles underneath. Then, when you swipe it, you are drag these particles along with the big lump of chalk that makes up your drawing. This is possible thanks to the internal cohesion of the material.

In the other areas of the board, you have reached a sort of steady-state with the current state of your eraser. This time there is a more direct contact with the surface of the eraser because there is no big lump of chalk that has been applied between it and the board. But the eraser is not clean and always leaves some amount of dust behind. This dust is not applied with the same pressure as with the chalk itself and is also too dispersed to form a cohesive lump, so that you are not able to drag all of it as a unit.