What is light, and how can it travel in a vacuum forever in all directions at once without a medium?

Well, I would say the electromagnetic field is the medium.

I don't understand this - How can something be the medium for itself?

Light isn't just an electromagnetic field. Light (i.e. photons) is a wave---a disturbance or perturbation---in an electromagnetic field. Just like for gravity, fields apparently permeate space. Even massive particles (e.g. quarks or electrons) can be described (or viewed) as waves in underlying fields.

So... maybe photons make up light, and they are just shooting through space like a wave, but it really isn't a physical wave (like a sound or water wave), it's just "acting like one"?

What you're calling a 'physical wave' is an emergent property of a underlying medium (e.g. air, water). For the most part, such an emergent wave is basically the same as a wave in a field. 'Physical waves' are often referred to as 'quasiparticles', because of this similarity. What we think of as 'particles' (e.g. electrons) don't just 'behave like waves', they are also waves, hence the 'wave-particle duality'.