What raster-based GIS systems really work?

I can't speak to SAGA or some of the other systems, but I have used GRASS extensively, including for an global-scale analysis of ~720M cells which required robust implementations of raster algebra and complex terrain operations. (As an aside, with the discontinuation of ArcInfo, GRASS is arguably the longest continuously developed GIS).

GRASS data and tools can be readily accessed through QGIS, which provides a nice ArcView GUI analog. QGIS itself is gaining nice raster analysis capabilities, such as the GDALTools plugin, but these are quite new and lack the maturity and depth of GRASS itself.

Another prospect is the raster package for R: R has a large userbase, the source of methods are easy to access, and it includes the cutting edge of many statistical techniques. However, it lacks in image processing tools and may not be sufficient for the kinds of tasks you're interested in.

Lastly, GDAL forms a solid basis of many, if not most modern GIS systems, and has very fast implementations of many common map algebra operations. It can be used through its Python interface or through direct C/C++ in times when the 'layer' abstraction proves insufficient.

We use a mixture - from Spatial Analyst, SAGA, Ermapper, a bit of GRASS, but in the end we tend to go to Geosoft - though that is because we do a lot of geophysical enhancement processing. Spatial Analyst/ArcGIS is good because you can easily extend functionality through the toolboxes/geoprocessing but we've found the actual Spatial Analyst processing routines are often not the best. Of late we've built toolboxes to access SAGA modules from within ArcGIS so we can continue to use the functionality without having to import/export - the toolbox looks after all that as necessary. We'll probably look at doing a similar thing to access GRASS functionality as well

You can now work and do map algebra with rasters of almost unlimited size in a spatial database with PostGIS. I personnaly work with SRTM and climate data at the scale of Canada. I can do intersection between raster and vector layers in a very fast and transparent way. I can also use a whole set of map algebra functions.