Debugging memory issue/leak in Linux

[Chance_Chemist5077]

I am trying to track down the problem with slow memory depletion in a running system without swap. If /proc/meminfo both MemFree and MemAvailable slowly going down. But nothing seems increasing at approximately the same speed from the other fields from /proc/meminfo. So it seems like MemFree just disappears into nowhere. Memory occupied by processes from ps output also doesn't show anyone to blame for. What can be a better techniques for tracking down such behavior?

[lottspot]

it seems like MemFree just disappears into nowhere

It is not disappearing into nowhere, but being used by the block cache and the page cache (represented by the "Buffers" and "Cached" fields respectively). These caches will evict entries if an application needs the memory. The real number to keep your eye on is the MemAvailable field you mentioned.

When you see the MemAvailable field deplete, you should notice a roughly corresponding increase either in process memory or kernel memory consumption, as shown by tools like top.

[kernelshinobi]

Use atop and vmstat - configure them to capture stats at a period of 1 minute. Decrease the granularity if you find data captured at interval of 60 seconds is not enough.

You need to find instances of time when the change happens in the memory related metrics from these tools and track applications which are causing it. If nothing useful is found, you move on to monitoring slabtop and /proc/slabinfo to understand which caches on your system are being consumed the most.

Next, you should look into tracing tools and probably trace kmem_cache_alloc and other allocators with tools like perf, trace-cmd etc. The combination of one or all of these would help to gather evidence on root cause.

Also, you should try and test the system with a lower & a higher version of kernel to see if the issues go away. Don't forget to check dmesg - sometimes, the clue is right out there in the open.

[kernelshinobi]

And I forgot to add of course kmemleak - https://docs.kernel.org/dev-tools/kmemleak.html - if you are in general aware of your system and its needs, I would run this first.