Remove a directory permanently from git

The ProGit book has an interesting section on Removing Object.

It does end with this:

Your history no longer contains a reference to that file.
However, your reflog and a new set of refs that Git added when you did the filter-branch under .git/refs/original still do, so you have to remove them and then repack the database. You need to get rid of anything that has a pointer to those old commits before you repack:

$ rm -Rf .git/refs/original
$ rm -Rf .git/logs/
$ git gc
$ git prune --expire 

(git prune --expire is not mandatory but can remove the directory content from the loose objects)
Backup everything before doing those commands, just in case ;)

Brandon Thomson asked in a comment to Rainer Blome's solution if this just fixed the gitk view or if the refs will be really gone. A good way to check this is to remember one of the sha1 hashes (or a unique prefix of it) of the old commits and try

$ git ls-tree hash-value

This should show you the content of the repos main folder as it was in this commit. After

$ rm -Rf .git/refs/original
$ rm -Rf .git/logs/

as shown by VonC and removing the refs/original/… lines from .git/info/refs and .git/packed-refs as shown by Rainer Blome, a final

$ git gc --prune=now

made not only the refs, but also the old objects (commits, trees, and blobs) go away. The above shown git ls-tree hash-value proves this. Another nice command to check this is git count-objects -v (run it before the filter-brach and after the pruning and compare the size).

Note: As I'm not allowed yet to comment on the other answers, I had to write a new one although it mainly combines previous given answers.

git-filter-branch by default saves old refs in refs/original/* namespace.

You need to delete them, and then do git gc --prune=now

Actually none of these techniques workedfor me. I found the most reliable was was to simply pull locally into another repo:

git pull file://$(pwd)/myGitRepo

It also saves you the hassle of deletig old tags.

see the story on my blog: http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/