Is there an NTP server I should be using when using Amazon's EC2 service to combat clock drift?
(2018) Amazon now recommend "just" using their 169.254.169.123 NTP server because
Your instance does not require access to the internet, and you do not have to configure your security group rules or your network ACL rules to allow access.
(It looks like the link-local "Amazon Time Sync Service" was introduced in late 2017)
Note: The 169.254.169.123 server does "leap smearing" and SHOULD NOT be mixed with other (non-Amazon) NTP servers from out on the internet that aren't doing the smearing exactly the same way. Amazon also recommend using chrony instead of ntpd unless you are stuck in a legacy situation where chrony is unavailable as compared to ntpd, chrony is faster at achieving synchronization, more accurate and more robust.
Yes, you should be using at least 3 and ideally 5 or more servers which are a low stratum and a close (round trip time) to your instance.
Amazon provide some documents which detail how to configure ntp. It should be noted that you don't need to use the pool servers listed - they are a front for the public ntp pool which Amazon load balance to; you can pick any servers you like, just remember to update your security/ACL settings for any new addresses.
The output you provided
[dalvarado@mymachine ~]$ sudo ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
173.44.32.10 .INIT. 16 u - 1024 0 0.000 0.000 0.000
deekayen.net .INIT. 16 u - 1024 0 0.000 0.000 0.000
dhcp-147-115-21 .INIT. 16 u - 1024 0 0.000 0.000 0.000
time-b.timefreq .INIT. 16 u - 1024 0 0.000 0.000 0.000
Shows that the servers you have configured are not reachable.
Refid=.INIT. means you have not yet initialised comms to the referenced server. You poll them every 1024 sec but they all have reach=0 thus you can't reach them and are not receiving the time from any server. That's why your clock is still wrong.
It maybe you have your firewall/network security setup too harsh and you are blocking access to those hosts, or more likely the port.
Do some network level diag as it would appear that's where your problem lies - also please include your ntp.conf and the output from ntpq -pcrv if you need further help.
Once you fix the reachability issue, check the numbers in ntpq -p are showing valid data and you should find your problem sorted and clock gets kept in check as expected.
Just a warning to folks about using the AWS time service at 169.254.169.123; This server is not a true ntp server as it doest not correctly handle leap seconds. Instead the AWS server does 'leap smearing'.
This may or may not be suitable for your setup, and you should never mix normal NTP and leap smeared NTP servers together in the same config, or the same timing domain. You should pick one standard and stick to it to avoid any problems.
Amazon documents NTP here. They include NTP configuration with their Amazon linux distributions. An Amazon instance that I have currently running lists these servers in /etc/ntp.conf, which is also what their documentation recommends:
server 0.amazon.pool.ntp.org iburst
server 1.amazon.pool.ntp.org iburst
server 2.amazon.pool.ntp.org iburst
server 3.amazon.pool.ntp.org iburst