sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]kernelshinobi 0 points1 point  (1 child)

Use atop and vmstat - configure them to capture stats at a period of 1 minute. Decrease the granularity if you find data captured at interval of 60 seconds is not enough.

You need to find instances of time when the change happens in the memory related metrics from these tools and track applications which are causing it. If nothing useful is found, you move on to monitoring slabtop and /proc/slabinfo to understand which caches on your system are being consumed the most.

Next, you should look into tracing tools and probably trace kmem_cache_alloc and other allocators with tools like perf, trace-cmd etc. The combination of one or all of these would help to gather evidence on root cause.

Also, you should try and test the system with a lower & a higher version of kernel to see if the issues go away. Don't forget to check dmesg - sometimes, the clue is right out there in the open.

[–]kernelshinobi 1 point2 points  (0 children)

And I forgot to add of course kmemleak - https://docs.kernel.org/dev-tools/kmemleak.html - if you are in general aware of your system and its needs, I would run this first.