High load average: Which processes are waiting in the queue?
You're seeing the unexpected loadavg because of high iowait. 98.7 in the wa
section of top shows this. From your screenshots I see the kworker process is also in uninterruptible sleep (state of D within top) which occurs when a process is waiting for disk I/O to complete.
vmstat
gives you visibility into the run queue. Execute vmstat 1
in typical sar
fashion for updates every second.
The r column shows runnable/running processes which the kernel uses to calculate loadavg and the b column shows processes blocked waiting for disk I/O aka uninterruptible sleep. Processes in b are added to the loadavg calculation, which is how iowait causes mysterious loadavg.
So to answer your question of how to see which procs are causing high loadavg, in your case of iowait, use top
/ps
to look for procs in a state of D then troubleshoot from there.
Linux, unlike most if not all other Unix like OSes, is not only counting processes using a CPU or waiting for a CPU in the run queue as a reference for its load calculation, but also add the number of processes (threads actually) being in uninterruptible state, i.e. waiting for for a disk or network I/O to complete. The latter are actually idle, i.e. not using the CPU.
There is then probably nothing to worry about your (not so) high load. The processes your are looking for are likely the single threaded redis
plus transcient kernel threads.