nvcc fatal : Value 'sm_20' is not defined for option 'gpu-architecture'

Try to change the code architecture (such as sm_20) to some higher version in CMakeLists.txt of stnbhwd that you are trying to install.

From:

IF (CUDA_FOUND)
   LIST(APPEND CUDA_NVCC_FLAGS "-arch=sm_20")

to:

IF (CUDA_FOUND)
   LIST(APPEND CUDA_NVCC_FLAGS "-arch=sm_30")

As you mentioned:

but I have Cuda 9.0

Cuda 9.0 doesn't support SM_20 architecture (more info).

You have quite new gpu:

I am running Ubuntu 16.04 on a GTX 1080ti.

which is based on Pascal architecture (SM_60). So you probably don't need the support for SM_20 architecture.