How do I stop villagers from moving away after telling them to?

You actually can stop your villagers from leaving after telling them they should leave!

According to Reddit user u/deenew, if you talk to the villager who is leaving on the same day that they told you they were leaving on a second player profile, you will get the "should I stay?" option again on the second player. Once you've told them not to go on the second profile, you can return on your resident rep profile, and they will inform you that they've had a change of heart and decided to stay.

Once you answer their question about moving or staying, the decision has been made. In the coming days that islander will pack up and leave. The only way to get them back is through amiibo cards, they come back to a campsite, or you find them again on a Nook island.