Where are Kubernetes' pods logfiles?
Short Answer:
If you're using Docker, the stdout
from each container are stored in /var/lib/docker/containers
. But Kubernetes also creates a directory structure to help you find logs based on Pods, so you can find the container logs for each Pod running on a node at /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/
.
Longer Answer:
Docker traps the stdout
logs from each container and stores them in /var/lib/docker/containers
on the host. If Kubernetes uses Docker as the container runtime, Docker will also store the containers logs in that location on the Kubernetes node. But since we don't run containers directly in Kubernetes (we run Pods), Kubernetes also creates the /var/log/pods/
and /var/log/containers
directories to help us better organize the log files based on Pods.
Each directory within /var/log/pods/
stores the logs for a single Pod, and each are named using the structure <namespace>_<pod_name>_<pod_id>
.
You can get the ID of a Pod by running
kubectl get pod -n core gloo-76dffbd956-rmvdz -o jsonpath='{.metadata.uid}'
. If you're used to usingyq
, you may find runningkubectl get pod <pod_name> -o yaml | yq r - metadata.uid
more straight-forward.
Within each /var/log/pods/<namespace>_<pod_name>_<pod_id>/
directory are more directories, each representing a container within the Pod. The name of these directories is equal to the name of the container. Lastly, when we look inside a /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/
directory, we'll find symbolic links to the log files stored by Docker inside /var/lib/docker/containers
.
Similarly, inside the /var/log/containers/
directory are symlinks to a /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/
directory. These symlinks are named using the structure <pod_name>_<namespace>_<container_id>
.
The on-disk filename comes from
docker inspect $pod_name_or_sha | jq -r '.[0].LogPath'
assuming the docker daemon's configuration is the default {"log-driver": "json-file"}
, which is almost guaranteed to be true if kubectl logs
behaves correctly.
This may also go without saying, but you must be on the Node upon which the Pod was scheduled for either docker inspect
, or sniffing around for the presence of log files on disk, to do anything helpful. kubectl describe pod $pod_name
will render the Node name, or as you might suspect it'll be in kubectl get -o json pod $pod_name
if you wish to acquire it programmatically.
Do you see anything in those directories?
In my clusters, the stdout/stderr logs from each pod are in /var/log/containers
, however there is some linking/redirection:
/var/log/containers/<pod-name>_<namespace>_<container-name-container-id>.log -> /var/log/pods/<some-uuid>/<container-name>_0.log
And that log is actually linked into /var/lib/docker
:
<container-name>_0.log -> /var/lib/docker/containers/<container-id>/<container-id>-json.log