Should I unmount a USB drive before unplugging it?
Either you're lucky to never have had corrupted data, or you're unlucky to never had noticed your data was corrupted.
When you perform an action that should write on a disk, most operating systems put the write operation into a queue. From time to time, they flush the queue. (I'm calling it a queue here, but actually operations can be performed out of order, operating systems do this when it's faster and gives the same final result.) This can make the write operations a lot faster, both because the system tries to perform them when it doesn't have anything better to do and because it can group them intelligently.
If you happen to unplug your device before everything has been written, you may miss the latest data. Worse, if the OS has been performing operations out of order, you may put your device into an inconsistent state and lose more than the latest data.
Some operating systems go into a more conservative (but slower) mode for removable devices, to reduce the risks associated with unplugging the device before it has been unmounted.
ADDED:
Doing operations out of order is sometimes not just a matter of speed. Cheap flash media (that doesn't to sector reallocation at the hardware level) has a limitation on the number of times you can write over any given sector. If you naively write all changes as they happen, this can kill the sectors that contain the file allocation table on a (V)FAT filesystem (the most common case for removable drives) or the journal on a typical modern filesystem. (See e.g. this discussion of sync on the Linux Kernel mailing list.) Here, not updating the FAT or journal every time a file is written to is not just a big performance gain, it's also good for the lifetime of the hardware.
Until recently, Linux only gave a choice between sync (write all changes as they happen) and async (write whenever it's convenient). Recent versions introduce the flush option for FAT filesystems, which is somewhere in between (flush all delayed writes as soon as the disk becomes inactive); it's on by default in Ubuntu 10.04.
On a different note, unmounting a removable drive ensures that no application has a file open. If you don't unmount before unplugging, you won't notice if you have unsaved data until it's too late. Unmounting while a file is open also increases the chance of corruption, both at the filesystem level (the OS may have queued some operations until the file is closed) and at the application level (e.g. if the application puts a lock file, it won't be removed).
The main risk you have is delayed writing. For various reasons the system doesn't always write the data to a disk when it is told to and keeps it instead in memory. When you unmount it, it makes sure all that is then written to the disk (and makes sure it isn't currently being used). You would probably know if you are currently writing to the disk, but you might not realize that your OS hasn't written everything you previously told it to write.
The frequency of this depends on your system and what your doing with the USB drive. If unmounts are usually slow and you can hear the writing noise coming from the drive, then you should probably continue to unmount. If unmounts are always instant though, then feel free to skip this step at your own risk.
I don't think anyone has addressed the issue of read vs write access. If you haven't copied anything to the flash drive, or opened a file on it for writing, you are probably safe just removing it -- if I only copy a file from a flash drive to my computer, I generally do not take the time to unmount it. But if I am copying files to the flash drive, then I do take the eatra step -- and yes, I have had corrupted files as a result of removing a flash drive too soon after writing to it.