Why is there a clear mirror image off a rough surface at shallow angles?

The reason for this is 2-fold:

1. Surfaces reflect better at almost parallel incidence

Let's focus on dielectric materials. Then this statement is a simple consequence of Fresnel's laws of reflection. Here are two reflection curves plotted against incidence angle from the wikipedia article (angle is measured from the normal to the reflection surface, so $90°$ means almost parallel to the surface):

As you can see in both cases (glass to air and air to glass) the reflection goes to one when the angle of incidence goes to $90°$.

This is fact is indeed used e.g. for X-rays, which essentially pass through everything at normal incidence, but are reflected well at "grazing incidence".

Here is an intuitive and not very precise reason for why this happens: when you shoot at something at almost parallel to the surface, then the component of the wave vector perpendicular to the surface becomes very small. So it has less energy perpendicular to the surface to punch through.

Just so that people don't kill me for the last sentence: this is sort of how Fresnel's equations are derived. The parallel wave vector is conserved and the perpendicular one transforms accordingly to match the frequency (energy) inside and outside the material. So that intuitive reason actually makes sense even though it sounds stupid.

2. Small incidence angles smooth over roughness

Imagine you look at the ocean. When you look from above, you will be able to see foamy wave tops all over the place. But when you look parallel to the surface, you will only be able to see one wave top. The other ones are all behind it, you might be able to see a few more a long way away, just peaking over the first crest.

That is intuitively why low angles smooth over rough surfaces. In a more rigorous manner we can state it in terms of the perpendicular component of the wave vector gain. Since it is smaller, the corresponding wavelength $\lambda = \frac{2\pi}{k}$ is much larger. The reflected beam can therefore only resolve much larger distance scales and roughness on small scales is smoothed over. I'd love to have a picture for this, but unfortunately I can not find a good one right now.