How do I append/prepend a timestamp to grep output?
a script to log […] every 5 seconds […] I need timestamps […] a loop […] the file gets to a certain size
You want a tool that takes as its input the log output of your main program, and that writes it to a size-capped log file, and adds timestamps. Tools exist that do this and more. The "more" that they do is automatic rotation of the log file that is also triggerable on demand, maintaining a size-capped log directory of current and old log files.
You have a choice of tools:
- Dan Bernstein's
multilog
from daemontools - Bruce Guenter's
multilog
from daemontools-encore - Laurent Bercot's
s6-log
from s6 - Gerrit Pape's
svlogd
from runit - Wayne Marshall's
tinylog
from perp - My
cyclog
from nosh
Assume a long-running utility that prints your desired log output every few seconds, something like:
#!/bin/sh # monitor-sensors exec 2>&1 while true do sensors sleep 5 done | grep --line-buffered -- '^Core'
Usage is as simple as:
monitor-sensors | cyclog ./temps
monitor-sensors | multilog t ./temps
and can be easily tweaked:
monitor-sensors | cyclog --max-file-size 32768 --max-total-size 1048576 ./temps
monitor-sensors | s6-log t s32768 S1048576 ./temps
monitor-sensors | multilog t s32768 n5 ./temps
From here, using toolsets like daemontools-encore/runit/perp/s6/nosh, it is but a small step to moving the left and right hand sides of this pipe into run
programs and running this as a linked pair of actual dæmons.
But they can all, also, handle being spun up for one-off output to existing log directories, if you just want to (say) perform one-off invocations from a command line.
Some of these tools can do other forms of timestamps, but they can all do TAI64N timestamps. TAI64N timestamps are designed to be capable of nanosecond precision, albeit that some of the aforementioned do not quite implement this; are designed to quite happily cope with things like leap seconds, as long as your TZ database knows about them; and are trivial to sort
, or even sort -m
.
Convert from TAI64N timestamps to your current timezone's local time (or, given that it's just the TZ
environment variable, an arbitrary timezone of your choosing) using tools such as:
- Dan Bernstein's
tai64nlocal
from daemontools - Bruce Guenter's
tai64nlocal
from daemontools-encore - My
tai64nlocal
from nosh
Watch such logs as they are written with:
tail -F
, albeit thattail
has known problems when there are very speedy rotations. (This is one of several known problems withtail
. Other known problems including handling in-place truncation of log files that are rapidly followed by more log data. This problem can be triggered by inferior systems such aslogrotate
. Fortunately, the aforementioned tools do not truncate files once they are fully written and do not risk such additionaltail
problems.)- My
follow-log-directories
from nosh, which "knows" this kind of log directory and uses a "cursor" (persistently maintained on disc) to reliably track the position in the log directory to continue from even if several rotations happen when the log follower isn't looking.
Other sorts of processing can be done with tools such as:
- Russ Allbery's
multilog-watch
logrange
- Paul Kremer's
multilog-stamptail
- My
export-to-rsyslog
from nosh
Further reading
- Jonathan de Boyne Pollard (2015). "Logging". The daemontools family. Frequently Given Answers.
- Bryan Cantrill (2013). How a pleasure cruise became an odyssey. OmniIT Surge 2013. YouTube.
- Bryan Cantrill (2012-07-29). Behavior of tail -f wrt truncation illumos-developers.
- https://unix.stackexchange.com/a/294206/5132
@JdeBP already answered what you should probably be doing. But that wasn't the question you made. So to anyone arriving here from Google trying to know how to append/prepend things to a multi-line output, here it is:
First, ^Core*
probably doesn't do what you think it does. It is a regular expression to match lines that start with "Cor" immediately followed by any number of "e"s: "Cor", "Core", "Coreeeee", etc.
Second, xargs
is a great tool, especially for one-liners and dirty-quick scripts. You can have xargs
to act on each line of input and issue a command for every one of them with the parameter -I
. So you can easily do something like this:
sensors | grep '^Core*' | xargs -I{} echo "${stamp}: {}" >> temps.log
Put '^Core*'
withing quotes because you don't want unexpected (though highly unlikely) shell expansions there. The {}
in -I
tells xargs
to replace {}
with each line of input in the arguments of the command it will be executing. You can choose another pattern, of course; I just like to mimic the syntax of find -exec
. The ${stamp}
assumes you saved the date/time in stamp
.
You can also raise a subshell, run the date
command (specifying whatever format you want) in it and have xargs
use its standard output as part of your arguments to echo
:
sensors | grep '^Core*' | xargs -I{} echo "$(date): {}" >> temps.log
Again, what you should probably be doing is using a tested and mantained tool for logging. But I'm answering here how to append/prepend a time stamp to grep output, not how to properly make your logs.