Recommended method to protect /downloader?

Just put a .htaccess (or if nginx/whatever a config) into the downloader directory with Disallow from all in it to forbid any request on the directory.

If you wanted to allow a few IP addresses in (like your own), try something like this in your .htaccess

order deny,allow
deny from all
allow from 1.2.3.4 5.6.7.8

Where 1.2.3.4 and 5.6.7.8 are IP addresses you want to let through.

My prefered way: Just delete downloader

Along with @daniel-sloof's recommendation I would say to ditch the Magento Connect installer altogether. I generally add it to .gitignore when setting up a new repository.

The reason being, as Fabian points out in his answer comments, that there is no way to ensure the replication of your production environment in source control without committing the packages from Connect. The feature you'll be losing here is the ability to update/upgrade packages from Connect - but if you really need that functionality you can always do it locally on your dev box and commit the results when you're satisfied that they work.

tl;dr:

Delete the /downloader folder or remove it from your source control.

I usually delete the downloader directory, but also found the following directive in the root htaccess helpful:

RewriteRule ^downloader/ - [L,R=404]

Which will make Apache send a 404 response even if the downloader directory is present.