Does driving a speaker with a DC offset AC signal matter?

Yes.

Figure 1. Cutaway view of a dynamic loudspeaker for the bass register. 1. Magnet, 2. Voicecoil, 3. Suspension, 4. Diaphragm. Source: Wikipedia.

A DC component in the signal will cause:

• Bias of the speaker cone (4) and the suspension (3) from the neutral position. This means that the cone will reach the limit of its travel - or at least where it becomes highly non-linear - in one direction in a shorter distance compared with the other.
• Maximum undistorted volume will be decreased as a result.
• The coil (2) will be heating even when no audio is playing.

There is no modern (read: solid-state, no transformer at the output) audio amplifier that doesn't leak at least some DC. It may be in mV range for a 100's W rated output at 4 or 8 ohm, but is measurable even when there is a capacitor at the output (it is usually electrolytic and usually leaks some dc).

Keep in mind that some high power low frequency speakers actually need the movement in order to cool its coil. A 100W rated "woofer" will probably burn from as low as 5W DC.

The other answers regarding wasted power and shortened travel of the cone are also pretty much true.

Edit: important to say, we are talking about the traditional speaker consisting of a coil glued to a cone and a permanent magnet. Piezo, for example, is way more tolerant to DC voltage (it still can be burnt or pushed outside of it's linear operation, but you have to try harder and apply a lot of DC, generally above the AC rating of the device).

Edit2: It is perfectly acceptable and widely used to drive a small and rarely-used speaker (like an interface beeper) with AC + DC (actually, turning it on and off with the frequency needed, so it gets either Vcc or 0). Just remember to leave it in "off" state when not needed or the the battery will go down. The speaker gets roughly Vcc/2 DC + Vcc/2 peak AC - as inefficient as it gets, but the beeping should be rare and short anyway. You can even use PWM or some A class amplifier and out some complex sound this way - that's how some kids toys speak and play music. In these cases, your "amplifier" is really simple - a single transistor connected to some GPIO of the controller.

This is also how the early wired telephones worked - generally, a speaker and a microphone (graphite, so it needs DC bias) connected in series. The DC bias for the microphone goes thru the speaker, but the speaker is designed to tolerate it. (Actually, no design efforts were ever made - these speakers operated well below their thermal limits and no one really cared about the sound distortion).