Is it safe to remove ~/.cache?

It is safe to clear ~/.cache/, new user accounts start with an empty directory anyway. You might want to log out after doing this though since programs might still use this directory. These programs can be found with this command:

find ~/.cache -print0 | xargs -0 lsof -n

In my case I would most likely be fine with just closing Firefox before removal.

Yes it is safe, but before you just throw it all away check what is in there. The files under .cache are used to speed up your programs e.g. by not causing downloads when not necessary. Some content there might immediately have to be downloaded or recreated again.

du -sm ~/.cache/*

will give you a list and you can selectively remove any of those subdirectories based on your usage of the programs, disc space used etc.

You could logout and back in as Lekensteyn indicated, but at most the programs owning the subdirectories have to be restarted if running.

It is safe, since by definition it only contains things that programs can recreate. As others point out it's probably a good idea to log out after clearing it.

A better question is perhaps: is it useful? It would fill back up again pretty quickly, unless you've significantly changed your usage patterns, and in the mean time would decrease the performance of the programs which use it while they recreate the information which was in there. You're probably better off leaving it alone.