How to stop playing with a player without being forced to quit and restart?

You click on the persons name who you want to end the game with then swipe left to right and click on the X .