From a quality perspective, what is better: turning volume up in the software, in the OS, or on the speakers?
Program vs. OS generally doesn't matter. What matters is whether you're adjusting volume in software or in hardware.
Reducing volume in software is basically equivalent to reducing the bit depth. In digital audio, the signal is split up into distinct samples (taken thousands of times per second), and bit depth is the number of bits that are used to describe each sample. Attenuating a signal is done by multiplying each sample by a number less than one, with the result being that you're no longer using the full resolution to describe the audio, resulting in reduced dynamic range and signal-to-noise ratio. Specifically, every 6 dB of attenuation is equivalent to reducing the bit depth by one. If you started with, say, 16-bit audio (standard for audio CDs) and reduced the volume by 12 dB, you'd effectively be listening to 14-bit audio instead. Turn the volume down too much and quality will start to suffer noticeably.
Another issue is that these calculations will often result in rounding errors, due to the original value of the sample not being a multiple of the factor by which you're dividing the samples. This further degrades the audio quality by introducing what's basically quantisation noise. Again, this mostly happens at lower volume levels. Different programs might use slightly different algorithms for attenuating the signal and resolving those rounding errors, which means there might be some difference in the resulting audible signal between, say, an audio player and the OS, but that doesn't change the fact that in all cases you're still reducing bit depth and essentially wasting a portion of the bandwidth on transmitting zeroes instead of useful information.
This PDF has more information and some excellent illustrations if you're interested in learning more.
The result of reducing the volume in hardware depends on how the volume control is implemented. If it's digital, then the effect is much the same as reducing the volume in software, so there's probably little to no difference in which one you use, in terms of audio quality.
Ideally, you should output audio from your computer at full volume, so as to get the highest resolution (bit depth) possible, and then have an analogue volume control as one of the last things in front of the speakers. Assuming all the devices in your signal path are of more or less comparable quality (i.e. you're not pairing a cheap low-end amplifier with a high-end digital source and DAC), that should give the best audio quality.
@Joren posted a good question in the comments:
So if I want to set software volume control to max, how do I deal with my analog controls suddenly having a super tiny usable range? (Because even turning the analog volume to half is way too loud.)
This can be a problem when the volume control is part of an amplifier, which is probably the case with most computer setups. Since an amplifier's job is to, as the name suggests, amplify, this means that the volume control's gain ranges from 0 to more than 1 (often much more), and by the time you've turned the volume control to the halfway point, you're probably no longer attenuating, but actually amplifying the signal beyond the levels you set in software.
There's a couple of solutions to this:
Get a passive attenuator. Since it doesn't amplify the signal, its gain ranges from 0 to 1, which gives you a much larger usable range.
Have two analogue volume controls. If your power amplifier or speakers have a volume or input trim control, that will work great. Use that to set a master volume level so that your regular volume control's usable range is maximised.
If the previous two aren't possible or feasible, simply turn down the volume at the OS level, until you've reached the best compromise between the usable range on the analogue volume control and audio quality. Keep individual programs at 100% so as to avoid several bit depth reductions in a row. Hopefully there won't be a noticeable loss in audio quality. Or if there is, then I'd probably start looking at getting a new amplifier that doesn't have as sensitive inputs, or better yet, has a way to adjust input gain.
@Lyman Enders Knowles pointed out in the comments that the issue of bit depth reduction does not apply to modern operating systems. Specifically, starting with Vista, Windows automatically upsamples all audio streams to 32-bit floating point before doing any attenuation. This means that, however low you turn the volume, there should be no effective loss of resolution. Still, eventually the audio has to be downconverted (to 16-bit, or 24-bit if the DAC supports that), which will introduce some quantisation errors. Also, attenuating first and amplifying later will increase the noise floor, so the advice to keep software levels at 100% and attenuate in hardware, as close to the end of your audio chain as possible, still stands.
Basically, in sound, the closer to the physical source the better, to have a clear signal. Each physical stage will add noise. Earlier, stronger.
When a signal is amplified, any noise in the signal will also be amplified. A stronger signal means less noise compared to the signal. Therefore as it gets passed down the chain there will be less noise.
Typically, I like to have my software levels and OS levels as loud as possible. Since these sources are generally not amplified, their decibel ceiling should be 0dB; Essentially, they can't clip.
I then make sure that this sound goes directly to a single amplified destination, such as digital headphones (via USB), speakers with a volume knob and power supply, or an amp. I try to avoid chaining amplified devices because they can start to overdrive each other and cause clipping. Even individually, the amplification can result in clipping if the volume is turned up too high.
Since these can clip, I tend to keep these sources around the 50% volume range since that's normally where they're comfortable. It also affords the flexibility of increasing or decreasing the volume if the software/OS level is lower than usual.