How do I remove/delete a virtualenv?

Just to echo what @skytreader had previously commented, rmvirtualenv is a command provided by virtualenvwrapper, not virtualenv. Maybe you didn't have virtualenvwrapper installed?

See VirtualEnvWrapper Command Reference for more details.

"The only way I can remove it seems to be: sudo rm -rf venv"

That's it! There is no command for deleting your virtual environment. Simply deactivate it and rid your application of its artifacts by recursively removing it.

Note that this is the same regardless of what kind of virtual environment you are using. virtualenv, venv, Anaconda environment, pyenv, pipenv are all based the same principle here.