How does Linux use a real time clock?
I can respond to some of these points, including the title.
[...] which always correlate with
systemd-timesyncdupdating the system clock. By this I mean that the very first syslog message after a time jump is aTime has been changedsyslog message:grep 'Time has been changed' /var/log/syslog Oct 2 23:53:33 hostname systemd[1]: Time has been changed
Actually, this message does not tell you what program caused the time jump. It is just a symptom of the time jump.
It happens when the kernel tells systemd the clock has been changed.[*] systemd responds by writing this message to the system log, and then recalculating when any .timer units will need to be triggered.
The message is printed by the program systemd, not by systemd-timesyncd.
More specifically, the message prefix "systemd[1]:" means it comes from process ID 1. PID 1 is the special "init" process. The systemd project also calls this the "system manager", to distinguish it from instances of systemd which manage user services.
The program called systemd does not change the clock after the system has finished booting.
In the current systemd source tree that you link to, the only program that even reads the RTC / hardware clock / hwclock is timedated, and only if you query it using timedatectl.
As I recall, older versions of the systemd program read the hwclock once at boot time, before running any other program, and set the system clock accordingly. In the latest version, systemd does not do this. There is only some hackery around telling the kernel which time zone is used for the hardware clock. (And avoiding triggering something very specific called "time warp").
In other words, current systemd appears to implicitly assume that something else initializes the system clock. In most cases, this will be the kernel.
Look up the kernel build option "Set system time from RTC on startup and resume" - CONFIG_RTC_HCTOSYS.
For a rounded understanding, notice there is also an option "Set the RTC time based on NTP synchronization" - CONFIG_RTC_SYSTOHC.
[*] System clock changes are detected using a Linux-specific feature. See TFD_TIMER_CANCEL_ON_SET.
Thanks very much to sourcejedi for this answer. This really led me to find the right answer.
Answer to the question
How does Linux use a real time clock to maintain the system clock?
It does so only once, during boot. It will not query the RTC again until the next reboot. This is configurable, but will do so by default on most kernel builds.
I want to know if an error with the real time clock would only present itself in the system clock when a timesync agent (ntpd or systemd-timesyncd) updates.
Unless the system is rebooted, the time in the RTC is unlikely to get into the system clock at all. Some agents like ntpd can be configured to use an RTC as a time source but this is not usually enabled by default. It's inadvisable to enable it unless you know the RTC is a very good time source.
Is there any direct link between the system clock?
It appears the time is copied the other way. The RTC is periodically updated with the system time. As per sourcejedi's answer, this is done by the kernel if CONFIG_RTC_HCTOSYS is set.
This can be tested:
Set the RTC
# hwclock --set --date='18:28'Then check the RTC time every few minutes with:
# hwclock
The result of this will be the system time will not change at all, and the RTC will eventually revert to the system time.
The cause of time jumps on the BBB
As sourcejedi pointed out, the messages were not being triggered by systemd-timesyncd. They were being triggered by connman. The evidence was (should be) a spurious log message in /var/log/syslog:
Oct 3 00:10:37 hostname connmand[1040]: ntp: adjust (jump): -27302612.028018 sec
...
Nov 21 00:07:05 hostname systemd[1]: Time has been changed
prior to version 1.37, connman is hard coded to promiscuously poll the default gateway for the time. It does not need to be DHCP configured to do this and if connman's NTP client is enabled (it is by default) then it will do this regardless of any other configuration.
In our case some home routers were actually responding to these NTP requests, but the results were highly unreliable. Particularly where the router was rebooted, it continued to hand out the time without actually knowing the correct time.
For example we know that at least one version of the BT Home Hub 5 will, when rebooted, default to 21st November 2018 and give out this date over NTP. It's own NTP client will then correct the problem but there's a window where it hands out 21st November 2018.
That is, this issue was ultimately caused by our customers rebooting their router and connman just accepting this time.
I'll express my frustration here, it seems the belligerence of some has left this "feature" in connman for far too long. It was reported as a problem as early as 2015. And it's a really well hidden "feature". There are no timeservers configured and no log message to explain what connman is doing or documentation as to why. If your test rigs have no NTP server on the default gateway you'll never see this in testing.
How to Fix
We are looking at two options which both appear to work:
Remove connman completely. It seems the network works just fine without it; we've not yet found the reason for it being there in the first place.
apt-get remove connmanDisable NTP in connman by editing
/var/lib/connmanto include:[global] TimeUpdates=manual