How do I remove a library from the arduino environment?

I have found that from version 1.8.4 on, the libraries can be found in ~/Arduino/Libraries. Hope this helps anyone else.

The answer is only valid if you have not changed the "Sketchbook Location" field in Preferences. So, first, you need to open the Arduino IDE and go to the menu

"File -> Preferences"

In the dialog, look at the field "Sketchbook Location" and open the corresponding folder. The "libraries" folder in inside.

Quote from official documentation as of August 2013:

User-created libraries as of version 0017 go in a subdirectory of your default sketch directory. For example, on OSX, the new directory would be ~/Documents/Arduino/libraries/. On Windows, it would be My Documents\Arduino\libraries\. To add your own library, create a new directory in the libraries directory with the name of your library. The folder should contain a C or C++ file with your code and a header file with your function and variable declarations. It will then appear in the Sketch | Import Library menu in the Arduino IDE.

To remove a library, stop the Arduino IDE and remove the library directory from the aforementioned location.

Go to your Arduino documents directory; inside you will find a directory named "Libraries". The imported library directory will be there. Just delete it and restart the Arduino app.

Your Arduino library folder should look like this (on Windows):

  My Documents\Arduino\libraries\ArduinoParty\ArduinoParty.cpp
  My Documents\Arduino\libraries\ArduinoParty\ArduinoParty.h
  My Documents\Arduino\libraries\ArduinoParty\examples
  ....

or like this (on Mac and Linux):

  Documents/Arduino/libraries/ArduinoParty/ArduinoParty.cpp
  Documents/Arduino/libraries/ArduinoParty/ArduinoParty.h
  Documents/Arduino/libraries/ArduinoParty/examples

The only issue with unused libraries is the trivial amount of disk space they use. They aren't loaded automatically so don't take up any application memory of the Arduino IDE.