Why does the light side of the moon appear not to line up correctly with the evening sun?

This is what you expect, in terms of the moon pointing towards the sun:

That is, the line across the moon appears perpendicular to the line towards the sun.

Now the above is a flat drawing. The sky appears curved (i.e. the dome of the starry sky). So that curve may introduce some apparent distortion.

To make a drawing that avoids the curvature issue, consider a drawing that only includes the sun, the moon, and a small amount of sky around the line connecting the two. (By the way, even for the curved bowl of the sky, the shortest line connecting the two is well defined except in the case of a new or full moon.) That drawing is approximately flat and will show the above relation.

I think the impression of not lining up correctly is due to our intuition, which is based on everyday scales of distances. Intuitively we expect the visible objects to be of similar sizes and at similar distances. But the Sun is much farther from both the Earth and the Moon, than the Moon is from the Earth: compare Sun-Earth distance of $1\,\text{AU}$ and Earth-Moon distance of $0.0026\,\text{AU}$ — it's $390$-fold difference! Thus although they form a triangle, the shading of the Moon doesn't seem to follow this, instead looking as if the Sun is infinitely far away.

Here's what you would normally expect based on our everyday intuition:

But if we take into account that the Sun is really large and really far away, we'll get something like this:

See how the shading of the Moon changed.

As an example, consider the following situation: the Sun is setting, and the Moon is in the opposite azimuth about $35^\circ$ above the horizon. You look at the Moon and see it as if it were lit from slightly above. This weirdness is simply because of the angle between your direction of sight and the light rays which light the Moon. Here's how it looks:

I think it is a parallax effect/optical illusion, and I'm not confident of explaining this clearly but here goes!

The normal vector to the illuminated portion of the moon is pointing generally away from the Earth/moon system towards a point over our horizon. At low altitudes (evenings) the sun will be close to the horizon and this can lead to the brain interpreting it as closer than it is and messing up the geometry. This is similar to the enlarged moon illusion when close to the horizon. Basically the normal vector appears to overshoot the sun as we interpret the sun as closer than it is.