Can I delete the folder "C:\WINDOWS\Installer\$PatchCache$"?

Heath Stewart's The Patch Cache and Freeing Space on MSDN blogs explains in detail. An excerpt:

[..] cause the Windows Installer service to cache any of the original files being replaced into the baseline cache. [..] It is this baseline cache that consumes a lot of drive space on the system drive after installing [..] The baseline cache facilitates patch uninstall by storing the original files so that they can be copied back to the target locations.

[..] by removing the baseline cache for a product, future repair, patch install, and patch uninstall scenarios may require your original installation media. If you have the drive space it is recommended that you keep the baseline caches available.

I think it should be safe to remove the files, provided you're able to download them again in future if they're needed or if you have access to your original install disks. I had something similar on my computer, I just deleted it and install things from the original disk every time it asks me now. Inconvenient, yeah; but I don't often install stuff and it's a lot better than having a whole ton of space on my hard drive wasted.

You could also try the MSIZAP utility from Microsoft. It has a "safe" option, 'G'.