How does the in-application programming for ARM (Cortex M3) work?

A common solution is to have a small reserved area in the flash, where the actual flashing program is stored. When new firmware has been downloaded just make a jump to the code in this area.

Of course, this small area is not overwritten when flashing firmware, it can only be done by other means (like JTAG). So make sure this flashing-program works good to start with. :)

I'm not familiar with STM implementation, but in NXP chips the IAP routines are stored in a separate, reserved ROM area which can't be erased by user code.

If you're implementing the flash writing code yourself by using HW registers directly, you need to either make sure that it doesn't touch the sectors it's running from, or runs from the RAM.