How can I tell what processes are causing high loads if they are not high-cpu usage?

Solution 1:

iostat can report statistics like that. Usually included in your distro in the package sysstat.

dstat might also be worth a look, it's a modern replacement.

Solution 2:

If you have IO accounting in your kernel, then you can use iotop to give information like that. Also, monitoring tools like collectd can record and report on the data.

---

Solution 3:

To find what's causing high load you can check few things.

• vmstat -w will show you ovierwiem (processes, swap, mem, cpu, io, system)
• pmstat -P ALL will provide you statistics (with %iowait) per cpu core
• iostat -x look for high %util or long await or big average queue size
  • dig deeper with iotop
• ps -ax look for state D which is uninterruptible sleep (usually IO), run it one more time check if they are still in D state
  • dig deeper with strace
  • check files and connections of those processes with lsof and netstat
• sar/sysstat - with that tool you can explore not just "now" stats but also check what was happening yestarday at midnight
  • sar -b - overall io activities
  • sar -d - individual block device io activities