Why does my system use more RAM after an hour of usage?

Unused RAM is wasted RAM. The Linux kernel has advanced memory management features and tries to avoid putting a burden on the bottleneck in your system, your hard drive/SSD. It tries to cache files in memory.

The memory management system works in complex ways, better performance is the goal.

You can see what it is doing by inspecting /proc/meminfo.

cat /proc/meminfo

You can reclaim this cached memory, using "drop_caches". However, note the documentation says "use outside of a testing or debugging environment is not recommended", simply because "it may cost a significant amount of I/O and CPU to recreate the dropped objects" when they are needed again :-).

Clear PageCache only:

# sync; echo 1 > /proc/sys/vm/drop_caches

Clear dentries and inodes:

# sync; echo 2 > /proc/sys/vm/drop_caches

Clear PageCache, dentries and inodes:

# sync; echo 3 > /proc/sys/vm/drop_caches

Note that sync will flush the file system buffer to ensure all data has been written.

From the kernel docs:

Page cache

The physical memory is volatile and the common case for getting data into the memory is to read it from files. Whenever a file is read, the data is put into the page cache to avoid expensive disk access on the subsequent reads. Similarly, when one writes to a file, the data is placed in the page cache and eventually gets into the backing storage device. The written pages are marked as dirty and when Linux decides to reuse them for other purposes, it makes sure to synchronize the file contents on the device with the updated data.

Reclaim

Throughout the system lifetime, a physical page can be used for storing different types of data. It can be kernel internal data structures, DMA’able buffers for device drivers use, data read from a filesystem, memory allocated by user space processes etc.

Depending on the page usage it is treated differently by the Linux memory management. The pages that can be freed at any time, either because they cache the data available elsewhere, for instance, on a hard disk, or because they can be swapped out, again, to the hard disk, are called reclaimable. The most notable categories of the reclaimable pages are page cache and anonymous memory.

In most cases, the pages holding internal kernel data and used as DMA buffers cannot be repurposed, and they remain pinned until freed by their user. Such pages are called unreclaimable. However, in certain circumstances, even pages occupied with kernel data structures can be reclaimed. For instance, in-memory caches of filesystem metadata can be re-read from the storage device and therefore it is possible to discard them from the main memory when system is under memory pressure.

The process of freeing the reclaimable physical memory pages and repurposing them is called (surprise!) reclaim. Linux can reclaim pages either asynchronously or synchronously, depending on the state of the system. When the system is not loaded, most of the memory is free and allocation requests will be satisfied immediately from the free pages supply. As the load increases, the amount of the free pages goes down and when it reaches a certain threshold (high watermark), an allocation request will awaken the kswapd daemon. It will asynchronously scan memory pages and either just free them if the data they contain is available elsewhere, or evict to the backing storage device (remember those dirty pages?). As memory usage increases even more and reaches another threshold - min watermark - an allocation will trigger direct reclaim. In this case allocation is stalled until enough memory pages are reclaimed to satisfy the request.

Memory Leaks

Now, some programs can have "memory leaks", that is, they "forget" to free up memory they no longer use. You can see this if you leave a program running for some time, its memory usage constantly increases, when you close it, the memory is never freed. Now, programmers try to avoid memory leaks, of course, but programs can have some. The way to reclaim this memory is a reboot.


You looked at a list of processes and their memory usage. But there was a problem. You were not looking at the complete list.

gnome-system-monitor only shows "My Processes" by default. To see processes owned by all system users, including the root user, click the menu icon in the top right (three dots in a vertical line). Change the selection from "My Processes" to "All Processes".

  1. Used v.s. available system RAM, as a single number
  2. Analyzing /proc/meminfo
  3. Other tools you could use - install atop today 8-)

1. Used v.s. available system RAM, as a single number

I see your ruby code subtracts MemAvailable from MemTotal. This is the exact same calculation used by gnome-system-monitor, when it shows the system is using "1.5 GiB (41.4%) of 3.7 GiB".

You are right to use either gnome-system-monitor or your manual calculation, at least as a first approximation. The MemAvailable number includes basically all the reclaimable "cache". I.e. MemAvailable includes a type of "cache" that is "available" to be reclaimed, as soon as a program asks for more memory than you have free.

Side note: There is another type or meaning of "cache", which is not reclaimable. When you look at the Cache / "cache" number, it is usually reported as including Shmem / "shared". The Shmem part is not reclaimable cache. The confusion happens because Shmem was cleverly implemented using the kernel "page cache".

Another way to quickly check "available" is free -h.

The free command also shows "shared", swap usage etc. Your system documentation should list the output fields and available options, i.e. in man free. Some of the other fields can be misleading:

  • The "used" field in the free command (currently) does not include "shared". This can be very confusing. Ignore the "used" field.
  • The "cached" value shown by free suffers from the problem noted above.
  • If free does not show "available", your system is ancient. Consult your ancient documentation.

2. Analyzing /proc/meminfo

Thank you for the full output of cat /proc/meminfo. This often helps to find specific answers (or any answer at all). If you wanted to look through how MemAvailable is calculated yourself, you could read the first section of my answer here: Is "Cached" memory de-facto free?

AnonPages

In your example meminfo, you have AnonPages: 924844 kB (0.9 GB). AnonPages is one of the terms that reduces MemAvailable.

When AnonPages is increasing, that should indicate an increase in the "RES", or "RSS" ("Resident" in ram "Set Size") of some number of running programs. But RSS can be misleading, because some memory is shared:

You cannot add up RSS, because that would double-count shared memory. You have to add up PSS, the proportional RSS after accounting for sharing. The smem command can show PSS, and also calculate totals. For example:

  • sudo smem -t > p; head -n1 p; echo; tail -n17 p - Look at per-process memory usage. The tail part will show the top 15 processes, followed by a line with total PSS etc.
  • smem -t -U ^sourcejedi$ > U; head -n1 U; echo; tail -n17 U - Look at memory usage of processes belonging to my user, "sourcejedi".
  • sudo smem -t -u - Look at memory grouped by user. This might be useful to distinguish your login session(s), from some system daemons that run as their own user. E.g. packagekitd runs as the root user, and can use hundreds of megabytes.
  • smem -t -P firefox - Look at the memory usage of my web browser :-).

  • sudo smem -t -m > m; head -n5 m; echo; tail m - Look at memory grouped by the mapping name - the name of the cached file, or "<anonymous>" or "[heap]".

    Process "resident" memory includes both "anonymous" memory, and some cached files. smem -m cannot show all cached files, only certain types of files that are currently in use. Specifically, files which a program has mapped into virtual memory. This includes program code, library code, and files mapped using mmap().

Shmem

You also have Shmem: 374848 kB (0.4 GB). I mentioned Shmem / "shared" above. This is another term that reduces the "available" memory. (It is not reclaimable cache). This is fairly normal, but you could try to see what it is.

Some shared memory is visible as the memory of individual processes. If the shared memory is mapped by a process, then it should be counted in the RSS / PSS. See above. The "mapping name" can sometimes be useful here (e.g. smem -t -m).

Shmem includes files on mounted tmpfs. You can check all mounted tmpfs using df -t tmpfs.

Depending on your system, Shmem could include some graphics buffers. I found a way to check their current size on my system (Intel graphics): Can I see the amount of memory which is allocated as GEM buffers? I would love to know if you find a different way to check on your system!

I read that Shmem memory leaks in some other graphics driver can be associated with very large VIRT (aka VSIZE) for Xorg. Linux using whole swap, becoming unresponsive while there is plenty of free RAM

Other memory usage?

  • MemTotal - MemAvailable = 1796088 kB (1.8 GB)
  • AnonPages: 924844 kB (0.9 GB)
  • Shmem: 374848 kB (0.4 GB)

Of the remaining 0.5 GB, I can see a few more small uses under 0.1 GB overall. The kernel also reserves a few percentage points of margin for itself (see "low water mark"), but I guess it would be 0.2 GB or less on your system. So there is a bit more usage that I am not sure about.

Kernel slab memory

"Unreclaimable slab" memory is another terms that reduces MemAvailable. You do not have very much: SUnreclaim: 55044 kB (0.05 GB).

You can also run slabtop to see the list of slabs. AFAICT, slabtop doesn't give stats on slabs as reclaimable or unreclaimable. But I can usually guess, and if there is some suspicious slab I guess you can look it up by name.

3. Other tools you could use - install atop today 8-)

smem might be overkill. Sometimes, all you need is top, or your favourite alternative, and to know how to sort by resident memory. (Although gnome-system-monitor might not be a good choice for this. I think it does not really show enough).

Sometimes when you have a performance problem, you need to look at disk reads & writes instead. You could use sudo iotop.

Sometimes, you would like to have a log of per-process memory usage, so you can look at why you ran out of memory and slowed the system to a crawl...

atop is a nifty little tool that can do all the above. If this sounds useful, I suggest you install it straight away. Then you can learn about it when you need it :-).

sudo atop -R shows "PSIZE" (same meaning as "PSS"). The atop package includes a background service that runs atop with a ten minute interval. You can use atop -r ... to open the log files, which are stored under /var/log/atop/.

Tags:

Performance

Ram

Recent Posts