Kubernetes - Liveness and Readiness probe implementation
Spring Boot 2.3 has built-in support for Liveness and Readiness Probes:
Spring Boot 2.3 has built-in knowledge of the availability of your application, tracking whether it is alive and whether it is ready to handle traffic.
In a Kubernetes environment, actuator will gather the Liveness and Readiness information and will expose it under health group.
"/actuator/health/liveness"
"/actuator/health/readiness"
For more details please check blog post and documentation
summary for documentation:
The Liveness state of an application tells whether the internal state is valid. If Liveness is broken, this means that the application itself is in a failed state and cannot recover from it. In this case, the best course of action is to restart the application instance. For example, an application relying on a local cache should fail its Liveness state if the local cache is corrupted and cannot be repaired.
The Readiness state tells whether the application is ready to accept client requests. If the Readiness state is unready, Kubernetes should not route traffic to this instance. If an application is too busy processing a task queue, then it could declare itself as busy until its load is manageable again.
Liveness Probe
Include only those checks which you think, if fails, will get cured with a pod restart. There is nothing wrong in having a new endpoint that always return an HTTP 200, which will serve as a liveness probe endpoint; provided you have an independent monitoring and alert in place for other services on which your first service depends on.
Where does a simple http 200 liveness helps?
Well, let's consider these examples.
If your application is a one-thread-per-http-request application (servlet based application - like application runs on tomcat - which is spring boot 1.X's default choice), in the case of heavy-load it may become unresponsive. A pod-restart will help here.
If you don't have memory configured while you starts your application; in case of heavy-load, application may outrun the pod's allocated memory and app may become unresponsive. A pod-restart will help here too.
Readiness Probe
There are 2 aspects to it.
1) Let's consider a scenario. Lets say, authentication is enabled on your second service. Your first service (where your health check is) has to be configured properly to authenticate with the second service.
Let's just say, in a subsequent deployment of your 1st service, you screwed up authheader variable name which you were supposed to read from the configmap or secret. And you are doing a rolling update.
If you have the second service's http200 also included in the health check (of the 1st service) then that will prevent the screwed-up version of the deployment from going live; your old version will keep running because your newer version will never make it across the health-check. We may not even need to go that complicated to authentication and all, let's just say url of the second service is hard coded in the first service, and you screwed up that url in a subsequent release of your first service. This additional check in your health-check will prevent the buggy version from going live
2) On the other hand, Let's assume that your first service has numerous other functionalities and this second service being down for a few hours will not affect any significant functionality that first service offers. Then, by all means you can opt out of the second service's liveness from first service's health check.
Either way, you need to set up proper alerting and monitoring for both the services. This will help to decide when humans should intervene.
What I would do is (ignore other irrelevant details),
readinessProbe:
httpGet:
path: </Actuator-healthcheck-endpoint>
port: 8080
initialDelaySeconds: 120
timeoutSeconds: 5
livenessProbe:
httpGet:
path: </my-custom-endpoint-which-always-returns200>
port: 8080
initialDelaySeconds: 130
timeoutSeconds: 10
failureThreshold: 10