View log files of crashed pods in kubernetes
kubectl logs
command only works if the pod is up and running. If they are not, you can use the kubectl events
command.
kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'
By default it does not sort the events, hence the --sort-by
flag.
Assuming that your pod still exists:
kubectl logs <podname> --previous
$ kubectl logs -h
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.
In many cases, kubectl logs <podname> --previous
is returning:
Error from server (BadRequest): previous terminated container "<container-name>" in pod "<pod-name>" not found
So you can try to check in the namespace's events (kubectl get events ..
) like @alltej showed.
If you don't find the reason for the error with kubectl logs / get events
and you can't view it with external logging tool I would suggest:
1 ) Check on which node that pod was running on with:
$kubectl get -n <namespace> pod <pod-name> -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME STATUS NODE
failed-pod-name Pending dns-of-node
(If you remove the <pod-name>
you can see other pods in the namespace).
2 ) SSH into that node and:
A ) Search for the failed pod container name in /var/log/containers/
and dump its .log
file and search for errors - in most of the cases the cause of error will be displayed there alongside with the actions / events that took place before the error.
B ) If previous step doesn't help try searching for latest System level errors by running: sudo journalctl -u kubelet -n 100 --no-pager
.