Ensure Kubernetes Deployment has completed and all pods are updated and available
Update #2: Kubernetes 1.5 will ship with a much better version of kubectl rollout status
and improve even further in 1.6, possibly replacing my custom solution/script laid out below.
Update #1: I have turned my answer into a script hosted on Github which has received a small number of improving PRs by now.
Original answer:
First of all, I believe the kubectl
command you got is not correct: It replaces all white spaces by commas but then tries to get the 4th field after separating by white spaces.
In order to validate that a deployment (or upgrade thereof) made it to all pods, I think you should check whether the number of available replicas matches the number of desired replicas. That is, whether the AVAILABLE
and DESIRED
columns in the kubectl
output are equal. While you could get the number of available replicas (the 5th column) through
kubectl get deployment nginx | tail -n +2 | awk '{print $5}'
and the number of desired replicas (2nd column) through
kubectl get deployment nginx | tail -n +2 | awk '{print $2}'
a cleaner way is to use kubectl
's jsonpath output, especially if you want to take the generation requirement that the official documentation mentions into account as well.
Here's a quick bash script I wrote that expects to be given the deployment name on the command line, waits for the observed generation to become the specified one, and then waits for the available replicas to reach the number of the specified ones:
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
deployment=
get_generation() {
get_deployment_jsonpath '{.metadata.generation}'
}
get_observed_generation() {
get_deployment_jsonpath '{.status.observedGeneration}'
}
get_replicas() {
get_deployment_jsonpath '{.spec.replicas}'
}
get_available_replicas() {
get_deployment_jsonpath '{.status.availableReplicas}'
}
get_deployment_jsonpath() {
local readonly _jsonpath="$1"
kubectl get deployment "${deployment}" -o "jsonpath=${_jsonpath}"
}
if [[ $# != 1 ]]; then
echo "usage: $(basename $0) <deployment>" >&2
exit 1
fi
readonly deployment="$1"
readonly generation=$(get_generation)
echo "waiting for specified generation ${generation} to be observed"
while [[ $(get_observed_generation) -lt ${generation} ]]; do
sleep .5
done
echo "specified generation observed."
readonly replicas="$(get_replicas)"
echo "specified replicas: ${replicas}"
available=-1
while [[ ${available} -ne ${replicas} ]]; do
sleep .5
available=$(get_available_replicas)
echo "available replicas: ${available}"
done
echo "deployment complete."