Apple - Where does iTunes save iOS updates?

On Windows XP
Documents and Settings\<User>\Application Data\Apple Computer\iTunes\iPhone Software Updates

On Windows Vista/Windows 7
Users\<User>\AppData\Roaming\Apple Computer\iTunes\iPhone Software Updates

On Mac
~/Library/iTunes/iPhone Software Updates

I believe that the updates do not delete after they are installed.

Via this iPhone FAQ article.

On Mac, you can delete the iOS firmware by going to Finder, pressing ⌘ cmd + ⇧ shift + G and go to ~/Library/iTunes/iPhone Software Updates. From there, you can delete the files.

You don't have to delete them unless you are short on disk space, as they get replaced by newly downloaded files whenever there is an upgrade.