Ingress gives 502 error
After a lot of digging I found the answer: According to the requirements here: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc#prerequisites the application must return a 200 status code at '/'. Because my application was returning a 302 (redirect to login), the health check was failing. When the health check fails, the ingress resource returns 502.
I just wanted to supplement the accepted answer with a more concrete explanation of why a health check on / is needed, even though livenessProbe and readinessProbe may be set up and working on the containers in the pods.
I had originally thought they were the same thing, but they're not.
The probes are used by the kubernetes engine to manage individual containers within a service. Whereas the health check on / is at the service level, and is part of the GCP load balancer contract. It's got nothing to do with kubernetes or containers per-se.
The reason it's needed with GKE is that the GCP load balancer is the default ingress controller. As stated in the docs, by default the GCP load balancer requires backing services to return a 200 on / to check if they're live so it can manage which ones to route to.
If you want to configure this health check endpoint:
- From GKE cluster version
1.17.6-gke.11(currently in beta) you can configure it on the GCP load balancer. See docs here. - Below
1.17.6-gke.11, you cannot configure it on the GCP load balancer. It has to be a 200 on/. The only option to configure it would be to use a different, more configurable ingress controller like the nginx one. I haven't used this myself, so can't point to any specifics on that.