How to get history of Pods run on Kubernetes Node?
We use Prometheus to monitor OOM events.
This expression should report the number of times that memory usage has reached the limits:
rate(container_memory_failcnt{pod_name!=""}[5m]) > 0
FYI: this is the next best thing to proper docs, the code
You can now use kube-state-metrics kube_pod_container_status_terminated_reason
to detect OOM events
kube_pod_container_status_terminated_reason{reason="OOMKilled"}
kube_pod_container_status_terminated_reason{container="addon-resizer",endpoint="http-metrics",instance="100.125.128.3:8080",job="kube-state-metrics",namespace="monitoring",pod="kube-state-metrics-569ffcff95-t929d",reason="OOMKilled",service="kube-state-metrics"}
Event history for your particular namespace, ordered by creationTimestamp:
kubectl get events -n YOURNAMESPACE -o wide --sort-by=.metadata.creationTimestamp
Or if you want to check the event history for all namespaces, ordered by creationTimestamp:
kubectl get events --all-namespaces -o wide --sort-by=.metadata.creationTimestamp