Ntpd on local network - Preventing clock drift causing high offsets

A driftfile records the frequency of the local clock, and is a good thing to have to accurately discipline it. Ubuntu /etc/ntp.conf has a driftfile by default.

Appending iburst to the server line sends a few packets rapidly on initial sync. Recommended, particularly if you do not want to wait several minutes for the first few packets.

Init scripts can delay starting the application until NTP is synced. Do this with a dependency on the ntp-wait systemd unit. Or the ntpwait script directly.

Your report of an offset growing and then correcting itself sounds like ntpd crossing its step threshold, then setting the time rather than slewing it.

Your lack of a relation between your NTP server's clock time and the real-world UTC time could be the root of your problem. If your NTP server does not have multiple real-world sources (regardless of whether they're out on the Internet or a GPS, PPS, or atomic source on your local network), then it will be unable to determine the frequency error of its own clock, and thus won't be able to discipline its clock effectively.

You can work out if this theory is correct by turning on stats logging on both server and client, like this:

statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

Then post your stats files and your configs to a pastebin or similar site and we can analyse them in light of the behaviour you're seeing.

After trying a few different options on ntpd still having a too big delay, I switched to chrony. My NTP server remained unchanged. Using chrony gave me a latency of <1ms all the time. I increased the polling time as recommended by chrony having following config file

### chrony.conf

# Add server
server 192.168.0.11 iburst minpoll -6 maxpoll -6 filter 15 xleave
hwtimestamp eth0 minpoll -6

# Logging
logdir /var/log/chrony
log measurements statistics tracking 

makestep 0.1 3
driftfile /etc/chrony.drift

And started the deamon with ./chronyd -f /etc/chrony.conf