rsyslog with logrotate: reload rsyslog vs copytruncate
Solution 1:
To answer your question, you first need to understand the different trade-off of reload and copytruncate:
- reload: the old log file is renamed and the process writing into that log is notified (via Unix signal) to re-create its log file. This is the fastest / lower overhead method: rename/move operations are very fast and have a constant execution time. Moreover, it is an almost atomic operation: this means that (nearly) no log entry will be lost during the move/reload. On the other hand, you need a process capable of reloading and re-opening of its log file. Rsyslog is such a process, so the default logrotate config use the reload method. Using this mode with rsyslog is strongly recommended by rsyslog upstream.
- copytruncate: the old log file is copied into an archive file, and then it is truncated to "delete" old log lines. While the truncate operation is very fast, the copy can be quite long (depending of how big is your logfile). Moreover, some log entry can be lost during the time between the copy operation (remember, it can be slow) and the truncate. For these reasons, copytruncate is not used by default for services capable of reloading and recreate their log files. On the other hand, if a server is not capable of reload/recreate log files, copytruncate is your safest bet. In other words, it does not require any service-level support. The rsyslog upstream project strongly advises against using this mode.
Solution 2:
Speaking as rsyslog author, copytruncate is actually a very, very, very bad choice. It is inherently racy and using it is almost a guarantee that you will lose log data. The more frequently the file is written to, the more you will lose. And this is not just part of the last line, but can be several hundred ones, depending on the exact timing and state of the system at the time the rotation happens.
When the file is moved and a new inode (file) created, rsyslog keeps track of the previous file and complete processing. So you do not have any loss in this case. Guaranteed (except if you unmount the file system...).
On "reopenOnTruncate": I personally have seen reopenOnTruncate to be racy in other regards as well, especially with NFS and the like. Some time ago I totally removed that functionality, but got later persuaded into merging similar functionality back in. It'll stay "experimental" most probably forever, as I really know people run into trouble on very heavily loaded systems. "copytruncate" is simply no decent mode to work with log files.
I currently work on refactoring imfile (ETA 8.34 or 8.35). The refactored version will probably be able to prevent accidental re-send due to API race, but also cannot guard against data loss - because this is conceptually impossible.
Solution 3:
This depends completely on how the process is writing logs.
copytruncate only works, if the log messages are appended to the file (e.g. whatever >> logfile.
And not when it is redirecting the output (e.g. whatever > logfile).