What happens to my boarding party if the enemy ship surrenders?

They stay in the enemy ship, but they stop attacking. You can move them around on the enemy ship, and (if you have sufficient sensors) you will still see all rooms inside of the opponents ship. Once you teleport them back, the internal ship view ends and it looks like a normal concluded battle.

It is amusing to have Lanius boarders: you can accept a surrender, and just leave your boarders in with the enemy characters until they asphyxiate. However this does not trigger any event or new rewards. The game considers the encounter over once you accept the surrender.

If I recall correctly; if you accept an enemy's surrender, your crew will be returned to your ship.

However, unless they are offering you something that you are badly in need of (crew / ship parts / etc) - if your boarding party was capable of taking out the crew without sustaining losses (and neither ship was in imminent danger of being destroyed); the scrap rewards for taking out the enemy crew without destroying the ship are fairly lucrative.