GRASS v.generalize method=douglas not working as expected
I finally got a workaround for the problem, based on Micha's suggestion (thank you!) to import the raster to GRASS and vectorize it there.
1 Import Raster:
r.in.gdal input=raster.tif output=raster
2 Vectorize (option -s leads to slightly smoothed 45-degree edges):
r.to.vect -s input=raster output=vector_blue feature=area
3 Generalize with Douglas to get rid of excessive points:
v.generalize input=vector_blue -c method=douglas threshold=0.05 output=vector_green
4 Smooth with Chaiken's algorithm to get smoother curves:
v.generalize input=vector_green -c method=chaiken threshold=0.1 output=vector_orange
5 Remove small polygons (area in m² if you have a lat/lon region):
v.clean tool=rmarea thresh=1000000000 input=vector_orange output=vector_red
If you can go back to the original raster, and import that into GRASS, then the module r.to.vect has a "-s" flag for smoothing the output vector boundaries. No generalize needed. Check this StackExchange question for a way to convert raster to vector in Arc and get smoother boundaries.