How do view older journalctl logs (after a rotation maybe?)

Solution 1:

It could be because you are trying to review the journal since the last boot, which seems likely to be the case inside a docker image.

On Ubuntu 16.04, the journal storage defaults to being in-memory. You can change the default to be persistent by opening /etc/systemd/journald.conf and changing the Storage= line from auto to persistent. You may need to restart journald by systemctl restart systemd-journald after the config file edit.

I think the journal should be persistent-by-default, so I opened a bug about that.

Solution 2:

The reason this happens is because of defaults on the size of journald files stored. There is more detail about this in the docs. It's worth reading the whole section I have linked to, but the defaults work like so:

journald will use 10% of the disk or 4G, whichever is smaller.

journald will leave free 15% of the disk or 4G, whichever is larger.

For viewing logs from the last boot, assuming you have Storage=persistent in your journald.conf, as the other answer notes, you can use the --boot=-1 flag on journalctl commands to get logs from just the previous boot.

In the case of the OP where they were sure the host had not been rebooted, the loss of logs was simply caused by the SystemMaxUse and/or SystemKeepFree defaults.

Note: I'm the OP and this question still has upvotes trickling in, so since I've gained more experience with journald (and rtfm) I am posting this here in the hopes it helps others.