How to uninstall downloaded Xcode simulator?

NOTE: This will only remove a device configuration from the Xcode devices list. To remove the simulator files from your hard drive see this answer.

For Xcode 7 just use Window \ Devices menu in Xcode:

Then select emulator to delete in the list on the left side and right click on it. Here is Delete option:

That's all.

You can remove them from /Library/Developer/CoreSimulator/Profiles/Runtimes (Not ~/Library!):