How to get history of Pods run on Kubernetes Node?

We use Prometheus to monitor OOM events.

This expression should report the number of times that memory usage has reached the limits:

rate(container_memory_failcnt{pod_name!=""}[5m]) > 0

FYI: this is the next best thing to proper docs, the code


You can now use kube-state-metrics kube_pod_container_status_terminated_reason to detect OOM events

kube_pod_container_status_terminated_reason{reason="OOMKilled"}

kube_pod_container_status_terminated_reason{container="addon-resizer",endpoint="http-metrics",instance="100.125.128.3:8080",job="kube-state-metrics",namespace="monitoring",pod="kube-state-metrics-569ffcff95-t929d",reason="OOMKilled",service="kube-state-metrics"}

Event history for your particular namespace, ordered by creationTimestamp:

kubectl get events -n YOURNAMESPACE -o wide --sort-by=.metadata.creationTimestamp

Or if you want to check the event history for all namespaces, ordered by creationTimestamp:

kubectl get events --all-namespaces -o wide --sort-by=.metadata.creationTimestamp