Apple - Sound always comes from Apple Thunderbolt Display even with headphones connected

In Windows, you can typically choose a different output-source using this approach:

For switching outputs, try this:
Right click on the speaker icon in the tray. Choose "playback devices". Select from the list of playback devices (speakers, hdmi, usb headset,) (the device you want) Choose "Set Default" Your audio output will instantly switch _^Source

For quick access to this, you can download this application.

Applicable to OS X users:
You should be able to see the earphones in the music menu. Hold down Alt while clicking the Audio-icon in the menubar. There you can switch between the different audio devices, like the external screen and the earphones.

This can also be done from System Prefrences -> Sound and the tab Output.

For those seeing this in OS X instead of Windows, there is a separate sound output selection box for "Sound Effects". Make sure to select Headphones.