large GeoTiff files on Geoserver

Based on the advice in "GeoServer on Steroids" I would aim for a mosaic of GeoTiffs. Page 8 clearly states to choose a mosaic when:

• A single file gets too big (inefficient seeks, too much metadata to read, etc..)
• Single granules can be large
• Use Tiling + Overviews + Compression on granules

but suggests moving to pyramids when

• Tremendously large dataset
  • Too many files / too large files
• Need to serve at all scales
  • Especially low resolution

So I would use GDAL to split them into a collection of GeoTiffs with tiles and overviews set and then drop them all into an Image Mosaic. Some experimentation could be useful to see what the best size of GeoTiff v. Number of granules is, but I'd err towards keeping it below a 100 if you are doing a lot of full views.

Update

To split up a GeoTiff you can use a script like:

#!/bin/bash
# If I was really bored I'd work out how to parse the gdalinfo output
#Upper Left  (     -25.000, 1240025.000) (  9d22'14.87"W, 60d50'27.75"N)
#Lower Left  ( -25.0000000, -25.0000000) (  7d33'24.37"W, 49d45'57.40"N)
#Upper Right (  660025.000, 1240025.000) (  2d48'16.84"E, 60d57'27.18"N)
#Lower Right (  660025.000,     -25.000) (  1d37' 1.96"E, 49d50'34.37"N)
#Center      (  330000.000,  620000.000) (  3d 6'26.52"W, 55d28' 7.68"N)

x=0
y=0
mx=700000
my=1300000
step=10000

for ((i=0; i < $mx; i+=$step)); do
  for ((j=0; j < $my; j+=$step)); do
    nx=$(($i + $step))
    ny=$(($j + $step))
    echo "gdal_translate -projwin" $nx $ny $i $j
  done
done

Ciao, an observation on number VS size of files.

Opening more than 10 to 20 file to respond to a request would hinder scalability of WMS and WCS. On the other side having Geotiff that are too large (hard to say but >> 20gb as a start) would make the internal structure too big for fast serving.

As such you need to balance size and # of files to read on each request. By experience in your case you could simple reprocess each file and then mosaic. If you want you might want to first merge to reduce the number of files to use. Use compression to reduce the size if possible.