What is the use of systemd-journal-flush.service?

The systemd-journal-flush.service asks the journal daemon to flush any log data stored in /run/log/journal into /var/log/journal, if persistent storage is enabled. In case you have (already) huge log files, this will result in slower booting. Further the disk (with /var/log) has to be mounted in a writeable modus to do so.

To sum it up: huge old log files, which are checked during boot and the appending of new log data results in slower boot time.

To check the journalctl log size type

journalctl --disk-usage

In order to get the time and disk space information of flush processing, enter the following command

journalctl -b --unit systemd-journald

The corresponding output will look like

-- Logs begin at Sat 2018-12-08 00:40:23 CET, end at Mon 2018-12-10 19:40:27 CET. --
Dec 10 12:51:38 ubuntu01 systemd-journald[479]: Journal started
Dec 10 12:51:38 ubuntu01 systemd-journald[479]: Runtime journal (/run/log/journal/265c93c062bf4c8da41abfe2ae793452) is 4.7M, max 38.3M, 33.5M free.
Dec 10 12:51:38 ubuntu01 systemd-journald[479]: Time spent on flushing to /var is 7.066904s for 132 entries.
Dec 10 12:51:38 ubuntu01 systemd-journald[479]: System journal (/var/log/journal/265c93c062bf4c8da41abfe2ae793452) is 128.0M, max 256.0M, 128M free.

You can either

• Disable the service (not recommended)

Then it is possible that not all log data is written to disk; annoying when debugging boot faults. Journald is a fundamental service in systemd linux and many other services depends on it.

• Verify the journal file for internal consistency:

   journalctl --verify
  

Look out that every journal shows up PASS.

• Use a journalctl --vacuum command

From journalctl -h

--vacuum-size=BYTES Reduce disk usage below specified size
--vacuum-files=INT Leave only the specified number of journal files
--vacuum-time=TIME Remove journal files older than specified time

Hence do a

    sudo journalctl --vacuum-size=1G --vacuum-time=5d --vacuum-files=5

• Change storage type of systemd-journal-flush.service

First check your storage type with

     systemctl cat systemd-journal-flush.service  | grep -i storage

From man journald.conf

Storage=

Controls where to store journal data. One of "volatile", "persistent", "auto" and "none".

If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed).

If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable.

"auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes.

"none" turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log buffer, or a syslog socket will still work however. Defaults to "auto".

Edit the file

    sudo nano /etc/systemd/journald.conf

In the journal section uncomment and alter:

    Storage=auto
    SystemMaxFileSize=1G
    SystemMaxFiles=5

Recommentation

I recommend to limit the size of the SystemMaxFileSize key to 20MB -->

    SystemMaxFileSize=50M

Finally, in case your Ubuntu is not running on an important server, I suggest to change the data storage to volatile:

    Storage=volatile

If you run the boot in debug mode and trace the system calls (strace) you may found out that the flush writing has a very poor i/o performance. In my case it was unclear why. Maybe some kernel messages spam the log file (note that after 10000 messages the unit is blocked per default, but journald has to manage this, which maybe cause the poor performance). In that case step over the messages and search for errors, which haven't necessarily marked as errors.

    journalctl -b --output short-monotonic

and

    journalctl -b -p 1..4 --output short-monotonic

The --output short-monotonicflag prints the time steps in contrast to the default utc time.

Finally remove the old log files by

    sudo rm -rf /var/log/journal

Save & reboot.

According to this post from the systemd developer homepage, you can fix it by changing the Unit file.

To do so, open /lib/systemd/system/systemd-journal-flush.service, eg

sudo vim /lib/systemd/system/systemd-journal-flush.service

and change the Before dependency from

 Before=systemd-user-sessions.service systemd-tmpfiles-setup.service

to

 Before=systemd-tmpfiles-setup.service

This fix will be automatically altered for systemd versions > v240.

Don't forget to save the file.