What keeps draining entropy?
Entropy is not only lost via /dev/{,u}random, the kernel also takes some. For example, new processes have randomized addresses (ASLR) and network packets need random sequence numbers. Even the filesystem module may remove some entropy. See the comments in drivers/char/random.c. Also note that entropy_avail refers to the input pool, not the output pools (basically the non-blocking /dev/urandom and the blocking /dev/random).
If you need to watch the entropy pool, do not use watch cat, that will consume entropy at every invocation of cat. In the past I also wanted to watch this pool as GPG was very slow at generating keys, therefore I wrote a C program with the sole purpose to watch the entropy pool: https://git.lekensteyn.nl/c-files/tree/entropy-watcher.c.
Note that there may be background processes which also consume entropy. Using tracepoints on an appropriate kernel you can see the processes that modify the entropy pool. Example usage that records all tracepoints related to the random subsystem including the callchain (-g) on all CPUs (-a) starting measuring after 1 second to ignore its own process (-D 1000) and including timestamps (-T):
sudo perf record -e random:\* -g -a -D 1000 -T sleep 60
Read it with either of these commands (change owner of perf.data as needed):
perf report # opens an interactive overview
perf script # outputs events after each other with traces
The perf script output gives an interesting insight and shows when about 8 bytes (64 bits) of entropy is periodically drained on my machine:
kworker/0:2 193 [000] 3292.235908: random:extract_entropy: ffffffff8173e956 pool: nbytes 8 entropy_count 921 caller _xfer_secondary_pool
5eb857 extract_entropy (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
5eb984 _xfer_secondary_pool (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
5ebae6 push_to_pool (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
293a05 process_one_work (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
293ce8 worker_thread (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
299998 kthread (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
7c7482 ret_from_fork (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
kworker/0:2 193 [000] 3292.235911: random:debit_entropy: ffffffff8173e956: debit_bits 64
5eb3e8 account.part.12 (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
5eb770 extract_entropy (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
5eb984 _xfer_secondary_pool (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
5ebae6 push_to_pool (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
293a05 process_one_work (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
293ce8 worker_thread (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
299998 kthread (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
7c7482 ret_from_fork (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
...
swapper 0 [002] 3292.507720: random:credit_entropy_bits: ffffffff8173e956 pool: bits 2 entropy_count 859 entropy_total 2 caller add_interrupt_randomness
5eaab6 credit_entropy_bits (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
5ec644 add_interrupt_randomness (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
2d5729 handle_irq_event_percpu (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
2d58b9 handle_irq_event (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
2d8d1b handle_edge_irq (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
230e6a handle_irq (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
7c9abb do_IRQ (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
7c7bc2 ret_from_intr (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
6756c7 cpuidle_enter (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
2bd9fa call_cpuidle (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
2bde18 cpu_startup_entry (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
2510e5 start_secondary (/lib/modules/4.6.2-1-ARCH/build/vmlinux)
Apparently this happens to prevent waste of entropy by transferring entropy from the input pool to outputs pools:
/*
* Credit (or debit) the entropy store with n bits of entropy.
* Use credit_entropy_bits_safe() if the value comes from userspace
* or otherwise should be checked for extreme values.
*/
static void credit_entropy_bits(struct entropy_store *r, int nbits)
{
...
/* If the input pool is getting full, send some
* entropy to the two output pools, flipping back and
* forth between them, until the output pools are 75%
* full.
*/
...
schedule_work(&last->push_work);
}
/*
* Used as a workqueue function so that when the input pool is getting
* full, we can "spill over" some entropy to the output pools. That
* way the output pools can store some of the excess entropy instead
* of letting it go to waste.
*/
static void push_to_pool(struct work_struct *work)
{
...
}
lsof is not the best tool to monitor /dev/random as one read by a process is over in a very short amount of time. I do not know of a good method of getting what process is doing a read, but using inotify you can monitor if there is a read.
Here there is basically two ways:
Get a summary after N seconds with:
inotifywatch -v -t 60 /dev/randomView live access events:
inotifywait -m --timefmt '%H:%M:%S' --format '%T: %e' /dev/random
Neither will give you process and the latter will not give you size of read. The first will give you a summary as in:
total access close_nowrite open filename
18 16 1 1 /dev/random
If you have that running and do a dd if=/dev/random of=/tmp/foo bs=1 count=3, you get the idea.
Anyhow. This will not give you ticks when the kernel consumes from the pool.
When it comes to checking the status of the entropy using
watch cat /proc/sys/kernel/random/entropy_avail
isn't the best idea as each cat is going to consume entropy. (I see now it popped up another answer that also mention this.) I also have some C code for this and tried to locate it yesterday. I'll see if I can find it and update answer later.