Is there an easy way to log all commands executed, including command line arguments?
Yes, there is a kernel facility: the audit subsystem. The auditd
daemon does the logging, and the command auditctl
sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve
system call:
auditctl -a exit,always -S execve
To specifically trace the invocation of a specific program, add a filter on the program executable:
auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool
The logs show up in /var/log/audit.log
, or wherever your distribution puts them. You need to be root to control the audit subsystem.
Once you're done investigating, use the same command line with -d
instead of -a
to delete a logging rule, or run auditctl -D
to delete all audit rules.
For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.
You could use snoopy.
Snoopy is more lightweight solution as it does not need kernel cooperation. All that is needed is dynamic loader (dl) that preloads snoopy library, path to which is specified in /etc/ld.so.preload
.
Disclosure: I am current snoopy maintainer.
The Linux kernel "audit" subsystem can do what you need.
e.g. if you run these commands:
auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve
Then every execution event is logged, and a lot of information is provided around that
e.g. this is the output of me running tail /var/log/audit/audit.log
exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null)
type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log"
type=CWD msg=audit(1543671660.203:64): cwd="/home/sweh"
type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1543671660.203:64): proctitle=7461696C002F7661722F6C6F672F61756469742F61756469742E6C6F67
There's some interesting values that can be seen; e.g. "auid" is 500, which is my login ID, even though "uid" is zero ('cos I'm running under su
). So even though the user may have switched accounts with su
or sudo
we can still track back to their "audit ID"
Now those auditctl
commands will be lost on a reboot. You can put them into a configuration file (eg in the /etc/audit/rules.d/
directory, on CentOS 7). The exact location will depend on your OS version. The auditctl
manual page should help here.
Beware, though... this will cause a lot of log messages to be generated. Make sure you have enough space on the disk!
If necessary the rules can be limited to a specific user, or a specific command.
And also beware; if a user puts the password in the command execution (e.g. mysql --user=username --password=passwd
) then this will be logged.