Generating contours from DEM using GDAL?
Cartographic rules to represent the relief as contours are presented in Imhof's famous book on relief representation, chapter C. Some of these rules are given on this wikipedia page. The main recommendation when simplifying contours it to preserve the terrain main characteristics.
Smoothing the contours independently do not prevent them to overlap: It is advised to smooth the DTM first. A traditional Gaussian smoothing with a suitable parameter depending on the target resolution allows to erase the small details. A drawback is that DTM smoothing fills the valleys and depressions, and flattens the ridges and peaks. Using the douglass-peucker filter algorithm like in this paper may be a solution. There are also number of methods based on the use of contour smoothing algorithms constrained by the drainage network or a skeleton. Finally, to prevent the contours to overlap in sloppy parts, it is possible to erase them locally or also deform them.