Apple - Connecting a Retina MacBook Pro to a VGA Display?

You should get a mini DisplayPort to VGA adapter (sometimes referred to as a Thunderbolt to VGA adapter, but it's the same thing). Apple sells one, but they can be had elsewhere for less, including at my personal favourite, Monoprice.

You could do HDMI to VGA, but it would require a (typically bulky) active adapter. There are some HDMI to VGA adapters that aren't active, but they rely on graphics card support that Macs don't have. The DisplayPort to VGA adapters are technically active adapters, but for whatever reason they seem to be much more compact than HDMI to VGA ones.

There are USB based display adapters, and some of them have VGA output, but I'd only recommend them for situations where other outputs aren't available. They're essentially a USB connected graphics card, which means that performance is lacking, and it takes up CPU and memory resources that the built-in outputs don't. Useful for some situations, but in your case I'd stay away.