TOTP Resynchronisation

Usually, the TOTP server is configured with an NTP server for accurate time. Each time the token authenticates to the server, the server adjusts for the drift of the token. As you have mentioned if a token is not used often, the amount of drift can surpass the synchronization window.

To solve this problem most of the time, TOTP servers provide a re-sync option. For example, using a software from the TOTP service provider installed on the user machine can allow the token to obtains the server time and re-sync its clock.

Here are some links to shows examples of the re-sync option

http://support.nordicedge.com/nsd1326-how-to-re-synchronize-oath-tokens-with-oathresyncwebapp/

https://kc.mcafee.com/corporate/index?page=content&id=KB76278

Some other TOTP service provider will do it manually through a help desk like http://www.mideye.com/support/users/token-card-users/

Of course, there are some other methods, but in general, all of them allow one way or another the token to know the server clock.