Generalizing polygon file while maintaining topology in QGIS?

GRASS is topology-aware. You can use v.generalize from the Processing toolbox to simplify polygons and if the input data is topologically correct so will the output.

This can be seen as a preliminary to @Underdark's answer whereby you can clean the topology of the vector layer before generalizing. GRASS has a v.clean function which contains a number of tools to repair the layer such as:

• snap which 'snaps' lines to the nearest vertex
• rmdangle which removes any annoying dangles
• rmdupl which removes duplicated geometry features

and various others shown in the image below:

You can also set the threshold depending on how sensitive you want the tools to run. Personally, I tend to use the Check Geometry Validity feature in QGIS (Vector > Geometry Tools > Check Geometry Validity) to check for any errors and if any exist, use the appropriate tool in v.clean. Once the errors are removed, you should be in a better position to generalize.

Hope this helps!

Have you tried using MapShaper? I've found this to be a good way of generalizing polygons; there's also an offline version if you want to download the code.

An alternative could be convert this to GeoJson, and then TopoJson which supports "topology-preserving shape simplification" https://github.com/mbostock/topojson