What is the safest way to remove a swap partition?

Try to monitor your swap memory usage in a terminal with top or htop, which is more user friendly. If you don't have htop installed, you can install it with:

sudo apt-get install htop

With your amount of RAM, I would say that your swap usage is 0% a lot of time.

If you want to format your swap partition, first you have to disable the swap with the swapoff command. Then format the partition with mkfs.ext4 or mkfs. After that you need to edit your /etc/fstab so that the system will mount your new partition at the boot time.

You can also use a graphical application like Disks, which ships with Ubuntu.

Having a swap partition on any kind of flash storage (like USB drives) is, indeed, not such a good idea.

It should be safe to simply remove the swap partition. While I personally never bothered removing it from /etc/fstab, it most certainly won't hurt either.

If you have sufficient amounts of RAM, the size of the swap partition becomes less relevant (please do correct me if I'm wrong, someone). I usually create a 4GB swap partition, and so far I can't say I've had any problems with it.
When the system is out of RAM (like, 0 bytes available), it will freeze. If it has a swap partition, it can move some data from RAM to swap to prevent the system from freezing. On the other hand, swap (HDD) is slower than RAM, so if it needs to fetch a lot of data from the swap partition, you're going to experience very noticable delays.