How to find the source of increasing memory usage of a twisted server?
As my guessing, it is due to memory fragmentation problem. The original design is to keep audio data chunks in a list, all of them are not in fixed size. Once the total size of the buffering list exceeds the limit of buffer, it pops some chunks from the top of list for limiting the size. It might looks like this:
- chunk size 511
- chunk size 1040
- chunk size 386
- chunk size 1350
- ...
Most of them are bigger than 256 bytes, Python uses malloc for chunks that are bigger than 256 bytes rather than uses memory pool. And you can imagine that those chunks are allocated, and released, what would happened? For example, when the chunk with 1350 size is released, then there might be a free 1350 bytes space in heap. After that, here comes another request 988, once malloc pick up the hole, and then there is another new little free hole of size 362. After long running, there are more and more little holes in heaps, in other words, there are so many fragments in heaps. The size of page of virtual memory usually is 4KB, those fragments are distributed around a big range of heap, it makes OS can't swap those page out. Thus, the RSS is always high.
After modification of the design of the audio chunk management module of my server, it uses little memory now. You can see the figure and compare to previous one.
The new design use bytearray rather than list of strings. It is a big chunk of memory, so there is no more fragmentation.
It does sound like a memory leak in a C module to me. Valgrind is a good tool to track memory-allocation related problems. I don't know how good it works with run-time loaded modules though...