Why do sunbeams diverge even though the sun is much more than a few kilometers away?

This picture (source) should pretty much answer your question:

The train's destination is not above the ground, but rather far away, and perspective means that the tracks appear not to be parallel but instead to converge to the vanishing point.

The same applies to the beams of light above them. The Sun is very far away and the beams are pretty much parallel, but they're pointing towards you, and perspective makes them appear to converge towards the vanishing point - which in this case is the Sun's location in the sky.

The technical term for these beams is "crepuscular rays." Occasionally, when the Sun is very low on the horizon, you can see "anticrepuscular rays," where the beams seem to converge to a different point on the opposite side of the sky to the Sun. Here's an example: (source)

This happens for the same reason - the rays are really parallel, and there's another vanishing point in the opposite direction from the Sun.

In addition to the excellent answers already given, here is another famous picture of an anticrepuscular ray (or better: shadow). It is not a coincidence that the shadow of the plume extends almost exactly towards the full moon. It is a consequence of the Sun, Earth and full Moon being almost aligned. So all shadows must point towards the moon at such a moment.

Just imagine what a picture would look like with a second shuttle starting a bit more towards the right... a "V" shape with the Moon at the corner...

Image credit: NASA, Pat McCracken.

This also was once the Astronomy Picture of the Day, where you can find some extra explanation.