Best way to manage large number of GeoTIFF rasters?

I would say then that your best bet would be to use a WMS (something like MapServer or GeoServer, or their commercial equivalents), and serve your image data as tiles. It'll take a bit of setting up, but it would avoid having to load all the files just to see a small area at a time.

There are plenty of questions and good answers here about setting up a WMS, as well as plenty of tutorials on the web.

We had this same issue until we decided to make it a Tiled Map Service and cached the tiles. The suggestion to use a WMS was a very good one, although taking it a step further making it a GeoWebCache seems like a pretty attractive option. There are also many optimizations that can be made on Web Mapping Services(WMS) to improve performance also if your first implementation didn't perform as well as you had hoped.

Also, there is a community wiki that you can reference for map service software options.