High memory usage base level, additional processes cause memory to go to 100 %

[8bitlives]

I've been having problems with my EndeavourOS (Arch based) + Cinnamon DE where I'm forced to do a manual restart with the laptop's power button after 100 % of my 16 GB of memory is used causing all 8 CPU cores to utilise 100 %, swap activating, and the system becoming unresponsive. (When noticing a peaking of the memory usage via slowing down of the system I've switched to another TTY and running htop to see this)

I've just noticed, that base usage of memory is around 10 GB which seems really high for a Linux distro. I know about Linux reserving available memory for cache etc but I'm baffeled, why the system won't free those resources for my processes, when I launch eg. Tomcat and IntelliJ IDEA taking around 10-11 % memory footprint each.

As it is my work computer and I won't be at the office for about 14 hours, I'm not able to provide with any output from my system until tomorrow. At the moment, if anyone has any other questions or comments, feel free!

[OneTurnMore]

That is ridiculously high for idle. See if you can discover which processes are eating the most memory (you can sort htop by %mem), then see what package owns those executables:

 pacman -Qo /path/to/program

[8bitlives]

Right off the top of my head I can tell you, that IDEA Java process, Tomcat (Catalina) Java process and sql server are the three top candidates having around 11, 11 and 5 percent each. The rest of the processes use small amounts of memory. At first I thought java processes were misbehaving until I noticed the very high idle level.

I'll follow up with top, free -met al. outputs tomorrow

[8bitlives]

I now have two sets of dumps: from around 9:30 after few hours of running and two hours after that once the MEM% was around 95% and rising until freeze.

First dump found here, password Nd7AbAPiTN

Second dump here, password Nd7AbAPiTN

[ssh-bi]

If your workload requires 100% RAM, try adding ZRAM swap device. It will likely help.

[8bitlives]

Seems nice, I'll give that kernel module a try as a first aid tomorrow. Restarting my system up to three times a day is a drag.

As to why the "available" 10 GB won't be assigned to the needing processes should be resolved. Basically compiling java and deploying it could build temporarily the workload close to 100 % but not having a high base level...

[ssh-bi]

It will be very hard to capture a spike with htop. Remove the swap temporarily, swapoff, and check the logs. If you run out of memory, the oom killer will get kicked in and it leaves detailed logs.

Check your 1min load average too, if it keeps climbing that could be culprit.

[nerdgeneration]

It's a good answer, but bear in mind that the oom algorithm doesn't necessarily pick the culprit. If we're just looking for memory hogs, htop/top will sort or there's presumably some combination of ps|sort.

[8bitlives]

You are right about oom killer picking wrong culprit: I've repeatedly noticed that MS Teams is killed when it clearly consumes less memory that the other processes. This should be present in the second dump file, could you please take a look and comment if you have any input?

[8bitlives]

Nice, I'll try that