How to customize error pages served via the default backend of an nginx ingress controller?

Alright, some parts of these answers were helpful on the hunt for the complete solution, especially the one from @Matt. However, it took me quite some time to get this working so I've decided to write my own answer with all the necessary details that others might struggle with as well.

---

The first thing would be to create a Docker image server capable of responding to any request with 404 content, except /healthz and /metrics. As @Matt mentioned this could be an Nginx instance (which I've used). To sum up:

• /healthz should return 200
• /metrics is optional, but it should return data that is readable by Prometheus in case you are using it for k8s metrics. Nginx can provide some basic data that Prometheus can read. Consider this link if you would like to get the full Prometheus integration with Nginx.
• / returns a 404 with your custom HTML content.

Thus, the Dockerfile looks like this:

FROM nginx:alpine

# Remove default NGINX Config
# Take care of Nginx logging
RUN rm /etc/nginx/conf.d/default.conf && \
    ln -sf /dev/stdout /var/log/nginx/access.log && \
    ln -sf /dev/stderr /var/log/nginx/error.log

# NGINX Config
COPY ./default.conf /etc/nginx/conf.d/default.conf

# Resources
COPY content/ /var/www/html/

CMD ["nginx", "-g", "daemon off;"]

In the same folder where Dockerfile is located, create this default.conf Nginx configuration file:

server {
    root /var/www/html;
    index 404.html;

    location / {

    }

    location /healthz {
        access_log off;
        return 200 "healthy\n";
    }

    location /metrics {
        # This creates a readable and somewhat useful response for Prometheus
        stub_status on;
    }

    error_page 404 /404.html;
    location = /404.html {
        internal;
    }
}

At last, provide a content/404.html file with HTML/CSS to your own liking.

Now build the Docker image with:

docker build --no-cache -t custom-default-backend .

Tag this image so that it is ready to be pushed into DockerHub (or your own private docker registry):

docker tag custom-default-backend:latest <your_dockerhub_username>/custom-default-backend

Push the image to a DockerHub repository:

docker push <your_dockerhub_username>/custom-default-backend

Now comes the part of integrating this custom-default-backend image into the Helm installation. In order to do this, we first need to create this k8s resource file (custom_default_backend.yaml):

---
apiVersion: v1
kind: Service
metadata:
  name: custom-default-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - port: 80
    targetPort: 80
    name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-default-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: custom-default-backend
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: custom-default-backend
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - name: custom-default-backend
        # Don't forget to edit the line below
        image: <your_dockerhub_username>/custom-default-backend:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80

Assuming we have a k8s namespace ingress-nginx already created we can create these two resources.

kubectl apply -f custom_default_backend.yaml

Now in order to tie the Nginx Ingress Controller with our new service, we could probably just edit the deployment of the Ingress Controller. But I've decided to remove it completely via Helm:

helm delete nginx-ingress -n ingress-nginx

And install it again with this command (make sure you have the --set flag with proper arguments included):

helm install nginx-ingress --namespace ingress-nginx stable/nginx-ingress --set defaultBackend.enabled=false,controller.defaultBackendService=ingress-nginx/custom-default-backend

With these steps you should end up with a working custom default backend implementation. Here is a GitHub repo with the files that I have used in this answer.

The project provides the Go custom error application that can be built into a container image to replace default-backend. The errorHandler function does the magic.

In the end it's a web server that responds to any request with 404 content, except /healthz and /metrics. You could do it with an nginx instance and html error pages if you want.

You probably don't want to use the full custom error handling, this is slightly different where the ingress controller will look for certain HTTP status codes from a regular app backend, and pass them to the default backend for handling. This causes issues for most application unless they were designed to use this from the outset.