How do I remove a directory from github?

Try this:

git rm -r vendor/plugins
git commit -m "Your comment here"
git push -u origin master

First deletes your directory
Second makes your commit
Third push your changes

You could try git rm -rf vendor/plugins to recursively delete it. You will need to commit this again.