Why don't household mirrors scatter light in all directions?

The effect of the roughness of the surface on the scattering of light depends, according to the Rayleigh criterion for rough surfaces, on the wavelength of light and on the incident angle.

Basically, it says that if the roughness of a surface, which could be characterized by the difference in height at peaks and valleys of the surface, $\Delta h$, is smaller than $\frac {\lambda} {8cos\theta}$, the surface could be considered smooth. It has to do with constructive and destructive interference of of the reflected light.

This is a very crude definition of roughness, but it gives you an idea that the same surface will appear smoother or more specular as the wavelength and the incident angle of the light increases.

So, although at the microscopic level the surface of a mirror may look rough, for the wavelengths associated with visible light, it is obviously pretty smooth and produces almost perfect specular reflection.

Why should it have a fractal structure? It's true that the surface isn't perfectly smooth, but it's very close. The roughness might be up to 10 nm or so, but visible light is ~550 nm, so it in effect doesn't really see the surface roughness.