Something eats all memory (I suspect memory leak on some app). How to detect what?

Solution 1:

Running top in batch mode to report memory sizes periodically can be used to see who is using the memory when things go south. Runing sar in batch mode should give some good diagnostics on memory use, and related I/O. Running munin to monitor the system should give you a graph with good detail on what memory is being used for. This may help a lot.

You can use limits.conf to limit the maximum core size of programs. Properly set, this should kill any programs which are leaking memory. This works with the pam_limits module. Limits can also be set with the ulimits command.

You are running a few programs which could use large amounts of memory. Some things you could look at include.

• Poorly programmed applications running under apache2 can leak memory. You should see the memory size increase when this happens. You can tune apache2 to recycle children after a certain number of uses by setting MaxRequestsPerChild to 100 or so. If this resolves the problem, then you need to resolve the leak. I would watch this first.
• MySQL may try to load data into memory. If you have a lot of data in memory this may cause some thrashing, but should not be as dramatic as you are seeing.
• If you have a large tmpfs file system mounted, then you may leak memory if files are not deleted when used. Large long lived files can also be a problem.
• If the problem is occurs at roughly the same time of day, you may have a scheduled program which is leaking memory.
• If you have a program that allocates shared memory, but does not release it before exiting, you will have a relatively invisible memory leak. If the shared memory is locked in memory, then it may force swapping. The amount of available shared memory is typically relatively limited.
• The liquidsoap+icecast bundle could run into buffering issues that use memory. I haven't used this combination, so I am not sure how this would appear.

Normal memory usage: Free memory is not something you want a lot of. If your system has been up for a long time and has a lot of free memory something is wrong. Every time you read or write a file, the blocks will go into the buffer cache. This will reduce your free memory, and is a good thing. The system will keep enough free space to start a few programs without looking elsewhere for memory. As many programs run quickly, their memory will be returned to the free pool when they stop running.

When you read a file that is in buffer cache, no disk access is required and the read is resolved from the buffer cache. Writes use a similar mechanism. If your system needs memory, the buffer cache is one of the first places that is used. Most buffers can be released immediately.

If you have a memory leak, you will see free memory and buffers both begin to shrink. This is still not a severe problem, as the leaked memory should eventually be moved to swap space. Your system will still run fine until you fill the swap space, and draw down the remaining free space to the point programs can't be started. It is typical that a small amount of swap space may be used.

Solution 2:

You can use this command to see the top 10 applications regarding RAM usage:

ps -A --sort -rss -o comm,pmem | head -n 11

Sometimes this command helps you if many sub processes have been generated:

ps auxf

This way you can see which processes belong together.

---

Solution 3:

Nothing is really using that memory in terms of applications.

You need to deduct the 'cached' value which represents the page cache to get a better idea as to what your actual memory usage is in terms of program usage.

Basically this is good memory management and this is ideally what you want.

See the link here for more infomation: http://www.linuxatemyram.com/