Creating transparent hillshade?

Nutshell

Each set of 3 images below should be read such as "grey (band) + opacity (band) = transparent result". You can test these processes within minutes via the associated github hosted makefile. Process #3 is the one which I recommend, with a threshold between 170 (keeps strong shadows) and 220 (keeps all shadows). Process 3 provides the strongest shadows and avoid greying-whitening effect. Adapt the resulting layer's overall opacity as needed. The equations in --calc="<equation>" may be improved as needed as well, using gdal_calc.

For a laid back video on this approach, explained by a Photoshop designer, see Adding Shaded Relief in Photoshop (16mins).

Background

gdaldem hillshade produces a one band grey scale file with pixels values range=[1-255], aka from the darkest shadows to the most enlighten pixel. For flat areas, px=221 (#DDDDDD). NoDataValue pixels get default nodatavalue 0, also, the darkest black in input and in output is and should be 1. With no opacity band defined, opacity is 100%.

gdaldem hillshade input.tif hillshade.tmp.tif -s 111120 -z 5 -az 315 -alt 60 -compute_edges

We want to define and control a 2nd opacity band !

Objectives

We want one greyscale band -b 1, it's the hillshade. Out of gdal, it's a grey band with continuous range such as px=[1-255]. We can crop out non-relevant areas (#2), or blacken it to px=1 and rely on the opacity band (#3).

We want one opacity band -b 2, generally the invert of the hillshade or a related function of that. We can crop out non-relevant areas (#2). It must be a continuous range of opacities such px=[1-255], otherwhise there is no elegance.

gdal_calc can be use to both do math on pixels from input files A,B,C... and check boolean values such as A<220, which returns 1 (true) or 0 (false). This allow conditional calculus. If the condition is false, the related part of the equation is nullified.

1. Grey hillshade made transparent

The following provides a very good two-bands results with the standard gdal hillshade greys and whiter areas made increasingly transparent :

# hillshade px=A, opacity is its invert: px=255-A
gdal_calc.py -A ./hillshade.tmp.tif  --outfile=./opacity.tif --calc="255-A"
# assigns to relevant bands -b 1 and -b 2
gdalbuildvrt -separate ./final.vrt ./hillshade.tmp.tif ./opacity.tif

2. Optimization via pseudo-crop (-b 1 & -b 2)

2/3 of the pixels on -b 1 (greyscale) become invisible to bare eyes when the opacity -b 2 is added, yet, these pixels keeps various whiter -b 1 and low opacity -b 2 values. They can be made all white transparent [255,1] pixels, allowing a better compression rate:

# filter the color band, keep greyness of relevant shadows below limit
gdal_calc.py -A ./hillshade.tmp.tif  --outfile=./color_crop.tmp.tif \
    --calc="255*(A>220) +      A*(A<=220)"
# filter the opacity band, keep opacity of relevant shadows below limit
gdal_calc.py -A ./hillshade.tmp.tif  --outfile=./opacity_crop.tmp.tif \
    --calc="  1*(A>220) +(256-A)*(A<=220)"
# gdalbuildvrt -separate ./final.vrt ./color_crop.tmp.tif ./opacity_crop.tmp.tif
# gdal_translate -co COMPRESS=LZW -co ALPHA=YES ./final.vrt ./final_crop.tif

3. Further -b 1 optimization (crop + blacken)

Since we have an progressive opacity band -b 2 to rely on, we could make -b 1 pixels either white px=255 via 255*(A>220), or black px=1 via 1*(A>220).

gdal_calc.py -A ./hillshade.tmp.tif  --outfile=./color.tmp.tif \
   --calc="255*(A>220) + 1*(A<=220)"
# gdal_calc.py -A ./hillshade.tmp.tif  --outfile=./opacity_crop.tmp.tif \
#  --calc="  1*(A>220) +(256-A)*(A<=220)".
# gdalbuildvrt -separate ./final.vrt ./color.tmp.tif ./opacity_crop.tif
# gdal_translate -co COMPRESS=LZW -co ALPHA=YES ./final.vrt ./final.tif

This result shows stronger shadows.

Result

Create a transparent hillshade has for immediate objective to remove the plain's former grey areas and associated unwanted but ubiquitous greying-fading effect. The wished byproduct is an increased control over the final visual product. The process described remove most grey and all white pixels. The colorful background plain image will keep its chosen colors when overlayed by the transparent-to-black hillshades, only the shadowed areas will be darkened. Comparison of process #2 (left) and #3 (right) below.

Overview "Before vs After" :

Zoom, please notice the shadows (before vs after) :

Further optimizations

White areas: One may also wish to keep the most enlightened areas to increase the 3D feels. It would literally be the symmetric of this current approach with minor threshold changes, then a merge of both outputs via gdal_calc. The plain would be 100% transparent, the darkest shadows and lightest enlighten areas opaque.

Smoothing: The input hillshade may be pre-smoothed to get a better end result, see Smoothing DEM using GRASS?

Composite hillshade (How to create composite hillshade?).

Bumped hillshade is interesting as well (description)

Notes

• The flat area threshold in gdal hillshade output is px=221 (#DDDDDD = [221,221,221]), marking flat areas. Also, hillshade's px=221 divides the images between in-shadow slopes (A<221) and in-light slopes (A>221) pixels.
• A processing threshold at px=[170-220] as proven good, it keeps near 100% of the eyes-noticeable shadows, which themselves barely stand for 15-35% of the relief area.
• Filesize > Compression: final.tif out of #1, #2, #3 is ~1.3MB without compression, then ~0.3-0.16MB after compression, 80% saving !
• Filesize > cropping: From .326KB in #1, crop color & opacity (#2) get to 310kb, blacken color (#3) get to 160kb. Cropping effect on filesize is between 5~50% reduction with threshold at px=220 and my input.

Another way to get the same result of a non-grey canvas more suitable for combining with other layers is the 'combined' option in gdaldem.

It performs a slope and hill shade and combines the two in one operation. Areas of 0 slope are white. Areas of 90 degree slope are black for the slope shade, with some illumination added by the hillshade layer.

gdaldem hillshade -combined -compute_edges infile outfile.tif

Then use a multiply layer compositing mode to 'drape' this over other layers.

Standard/Combined hillshading

Combined shading multiplied with OSM base layer (opacity around 50%)

gdal + convert based workflow

There is a gdal + convert solution which gives good visual results. The trouble with this solution is that convert destroys geographic informations which you then have to restore. It increase the number of action to run.

# Basic crop
gdal_translate -projwin 67 35.92 99 5 ../data/noaa/ETOPO1_Ice_g_geotiff.tif crop_xl.tmp.tif
# Grey-based hillshade
gdaldem hillshade crop_xl.tmp.tif shadedrelief.tmp.tif -s 111120 -z 5 -az 315 -alt 60 -compute_edges
# create a transparent hillshade:
convert shadedrelief.tmp.tif -fuzz 7% -fill "#FFFFFF" -opaque "#DDDDDD"  whited.jpg # makes all grey values white to lighten the filesize
convert whited.jpg -alpha copy -channel alpha -negate +channel trans.png # <=== TRICK HERE.
# Restore georeferencing & reproject            
gdal_translate -a_ullr 67 35.92 99 5 trans.png trans.tmp.gis.tif
gdalwarp -s_srs EPSG:4326 -t_srs ESPG:3857 ./trans.tmp.gis.tif ./trans_reproj.tmp.gis.tif
# Compress from 11MB to 2MB:
gdal_translate -co COMPRESS=LZW -co ALPHA=YES ./trans_reproj.tmp.gis.tif ./trans.gis.tif

For command 4 explanation, see: https://stackoverflow.com/a/23018544/1974961