Why can we only "see" reflected light?

The key is that light must enter the eye for you to see something.

You cannot see a beam of light from a low powered laser which is not directed into your eye if the air through which the light is travelling is devoid of dust.
Adding dust to the air and you can see the trajectory of the laser beam because of the light being reflected/scattered from the dust and enters your eye.

Similarly no atmosphere on the Moon leads to a black sky even in daylight whilst on the Earth the sky is blue.

To see something light must enter the eye and the rods (and cones) must be stimulated sufficiently for the signals to be produced for processing by the brain.

The reflected light is moving toward/into your eye, while the light just passing by you isn't. You can see light that's not "reflected", like the light emitted by a light bulb, there's nothing special about reflected light. All that's needed to see light is the light actually hitting your retina.

The premise of your question is wrong - You can see light direct from a source!

This is what happens when you see the sun or a light bulb or a fire.

Most objects do not make their own light, they are illuminated by the sun or another source and we see them as a result of the sunlight reflected of them. If you go into a cave, you cannot see. Everything looks black because there is no sunlight.

Only the photons entering your eye enable you to see. Photons travel in straight lines, so you cannot see around corners. Your eye is tiny compared to the room you are in, so most of the photons bouncing off the objects around you go in other directions. Some of them may enter your friend's eye if they are with you.

So - what happens to the photons reflected off the objects around you that don't enter your eye? Well, if you are indoors, most of them will be absorbed by other objects and their energy will be dissipated as heat. If you are outside, quite a lot of them will be angled up towards the sky and out into space. Eventually, they may hit a planet or a nebula and be absorbed. It doesn't really matter.