Android - What are the differences between a system app and user app?
The other answers are correct, but fail to point out a couple differences:
Permissions
System apps are granted the ability to request certain system-only permissions that are never available to user apps. This is not the same as root/sudo/su permissions.
One example that I know of and use daily is the ability to reset the missed call notification counter. I have Go Dialer, which is a replacement for the stock dialer and contact app (how you make phone calls). When I miss a call, android lets me know that with a notification. However, Go Dialer is not allowed to reset this counter, even though it asks for that permission when you install the app. Android 2.2 and earlier let user apps reset it, but made this permission a system-only accessible permission for android 2.3
By moving my Go Dialer app to the system partition, it is allowed to reset this counter. Root is not required for the app (except, of course, to actually move the app to the System partition).
Another system permission is the ability to set the current gps location. Now, Android does have a preference where you can allow mock locations, so any gps spoofing app you install can declare a gps location. But if the app is in the System partition, mock locations can be disabled while still letting the app spoof the gps location.
Updates
System apps can be updated just like user apps, but the update is never integrated into the original rom. That is, system apps have the unique ability to roll back to the version that existed when the rom was first installed.
Move to SD
Android lets users move user apps from internal memory to the sd card. System apps cannot be moved and always take up some internal space. Most (all?) devices, though, have internal user apps on a different partition, so uninstalling a system app would not give you any more user app space (other than the dalvik-cache being removed).
You will recover some space if you uninstall any updates to a system app. As noted, the updates are not integrated with the rom and are thus stored in user space. The updates are not allowed to move to the sd card either, but if removed, user space is recovered.
How it affects you
If you are a casual android user, you likely won't run into scenarios where system vs user app would make a difference. If you are an advanced user, you likely have root and can thus change a system app to user (and vice versa) if you ever run into one of these situations.
/system is read-only without root so it prevents uninstalling applications from /system/app and /system/priv-app. Applications that are critical are put there so that they can't be uninstalled. Carrier bloatware that they don't want to let you remove is also put there. The latter category of apps can be removed if you are rooted; the former, not so much.
/system/priv-app also allows apps installed there to use signatureOrSystem and other privileged permissions. Stephen's answer has some examples.
As for why apps might be critical, that depends on the ROM. HTC's Sense is heavily dependent on their built-in apps from what I hear, for example. They simply didn't design their UI such that it would work natively with third party apps (or fail nicely if the system apps were missing). Other apps are essentially part of the Android OS but Google coded them as apps/services (probably for encapsulation and reliability reasons).
Updates for some system apps can be downloaded via the Play Store, such as Google Maps. If it didn't come preinstalled as a system app on your device then you can use Titanium to convert it from a user app to a system app.
System apps are apps that are included within the system. They are located in the /system/app folder.
The vast majority of them are required to make the device run. An example of some of these are the dialer, which sets up all the network connectivity, and browser, without which no in-app webviews would work.