Apple - Application level volume control in OS X?

A free and open-source solution is BackgroundMusic.

A nicer and paid solution is Rogue Amoeba's SoundSource

Detour is no longer under development, and doesn't work in MacOSX 10.5+

Hear is a rather expensive utility ($49.95) and has a number of other audio "Pro" features.

Jack is a low-latency audio server written originally for GNU/Linux and I believe can change volume on a per-application basis. And it's free.

SoundBunny ($9.99) also provides this function. Supports 10.6, 10.7 and 10.8

SoundBunny by Prosoft Engineering

Update: As pointed out below by Steve Moser SoundBunny cannot change the volume of Sandboxed applications.