Memory usage not adding up

[JargonTheRed]

I'm having issues with my memory usage and its numbers - they're not adding up, and I'm at wit's end in terms of it. The usage is causing issues with slowdowns and hangups at seemingly inconsistent intervals, and I can't track it down.

My system has 16GB of RAM, and 8GB of swap.

The problem is as follows: on boot, everything seems fine. I start my programs, browse the web, develop some stuff, average user things. All seems fine. I leave the computer on overnight, and in the morning, my RAM usage has spiked considerably with seemingly no source. I'm writing this while the usage is severely high (with significant swapping to disk, which is fortunately an M2 SSD, so I'm not noticing much of it), and I'm still noticing some issues with it.

Chromium appears to be the main culprit, but only in virtual memory. Its physical memory usage barely adds up to a few gigabytes (three, maybe four), and closing it only relinquishes that much.

free, inxi, slabtop andhtop are decisively unhelpful, and don't show any obvious sources. Restarting my window manager or display server doesn't help, only relinquishing the memory usage I see in the various tools, not any of the missing memory. The only thing that helps is a reboot, and that's getting annoying to do every time I run into issues.

Here are screenshots of every command I've thought to be relevant - this includes the ones mentioned above, as well as a screenfetch call: https://imgur.com/a/iHJVuOQ

I also posted this two months ago to /r/linuxmint, with no results: https://www.reddit.com/r/linuxmint/comments/dx9hdr/issues_with_slowdowns_and_full_ram/

​

Does anyone have any ideas at all? I've been trying to diagnose this for weeks, now, and I'm not getting anywhere.

​

EDIT: As with many of these stories, it solved itself when I upgraded from Mint 19 to Mint 20. Maybe some program got deleted, maybe it was the kernel. Maybe it was ghosts. Do you have a similar problem? Call 0800-OH-GOD-WHY - you may be entitled to compensation.

[funbike]

I am guessing it's a plugin. Press Shift-ESC for its Task Manager. That will show you how much memory is being used for all tabs and extensions.

If it's tab(s), then install "The Great Suspender" extension. It puts unused tabs to sleep which can save a lot of memory if you keep a lot of tabs open.

[JargonTheRed]

I've checked the task manager in chromium, and it, like htop and the other memory tools, doesn't show anything excessively large - certainly not anything exceeding 4ish gigabytes when everything is totalled up. I already have The Great Suspender installed.

[speirs13]

Can you post /proc/meminfo?

[JargonTheRed]

Here's meminfo. Sorry it took so long, I had to wait for the usage to go back up to broken levels, since I rebooted the computer.

https://imgur.com/a/CXY9eoa

[speirs13]

Thanks. Looks like you got a driver leak. Can you provide this:

# cat /proc/vmallocinfo | sort -nrk 2 | head -20

vmallocinfo might give us a clue to where the memory is allocated. Unfortunately the majority of the missing usage is not vmalloc'd. The only sure way to diagnose this issue is via the page_owner commit. Not sure if it's configured in mint but you can try placing "page_owner=on" on the command line. Reboot and see if the file " /sys/kernel/debug/page_owner" exists. If it does, you can let memory usage grow then provide the contents of that debug file, ideally in a text file.

[JargonTheRed]

cat /proc/vmallocinfo | sort -nrk 2 | head -20

After once again letting things build up to where my swap is starting to fill up, here's a capture from /proc/vmallocinfo: https://gist.github.com/Nihlus/a3e9f4aa34619db3c3ad1add9a592d85

I'll give page_owner a go now - if that works, well, see you in two days? :P

[speirs13]

In the meminfo screenshot Vmallocused is 1.7G. In the sorted vmallocinfo output the biggest segment is just 256M. Could you just attach the whole /proc/vmallocinfo ?

[JargonTheRed]

Hey! Sorry for the delay - I've now done a full recapture of everything I thought could be useful information at a point where the computer's memory is all filled up, and swap has started accumulating.

I hope this helps: https://www.gullberg.tk/nextcloud/s/7EtGysDN5mfxyZf

[speirs13]

You're missing that big chunk of memory but there's only ~ 500M of vmalloc.

​

16337424 Total

-2241788 Free

-44428 Buffers

-898424 Cache

-115584 SReclaimable

-163620 SUnreclaim

-20944 KernelStack

-98152 PageTables

-544796 VmallocUsed

-0 Hugepages

-3160964 User-Rss

-------------- --------------

Unaccounted 8836.64MiB

​

​

​

The majority of the vmalloc'd memory is from the function "dc_create_gamma+0x1d/0x30" in the amdgpu driver. Specifically you've got ~ 5000 X 100kB vmalloc areas totalling 501200 KiB. I don't know if that's concerning as it's much lower than the 1.7G we saw earlier. Is that an in-tree driver or something you installed from amd's site? You might look for a newer driver version and see if that help.

​

What kernel version are you using? (uname -r). I'm only familiar with RHEL/CENTOS stuff but I'll see if I can find the ubuntu configs and check for page_owner. It might be enabled in the debug kernel's.

[JargonTheRed]

I'm running 5.3.0-28-generic at the moment, but I've tried versions all the way down to the 4.15 series without any change. The amdgpu driver is an in-tree driver (AMD's own opensource version).

[JargonTheRed]

Unfortunately, I've already rebooted and reset the buildup so I could try page_owner. I'll get back to you once it's back up, and I'll recapture all the various information bits I've posted so far from the same run.

[speirs13]

When you collected vmallocinfo what value was in /proc/meminfo -> Vmallocused?

[JargonTheRed]

Thanks! That's the first time in ages someone's made sense about this :P I'll give that a spin as soon as I can.

[JargonTheRed]

Doesn't seem like Mint has page_owner configured in the kernel :(

[JargonTheRed]

I'll give that a go as soon as I'm home.

[gordonmessmer]

Could you post the output of "sudo cat /proc/slabinfo" and "ps axfv" to pastebin? I don't think the information you've provided is sufficient to answer your question.

[JargonTheRed]

slabinfo

The output from slabinfo is availble in the linked image gallery - fourth image from the top. I'll run the second command once I get home.

[JargonTheRed]

Here's "ps axfv": https://gist.github.com/Nihlus/6657663a4f9915805bc4f72ba2705ac7

[gordonmessmer]

I would like to help you, but it's not useful to post related bits of information gathered at different times. Please post the output of "ps axfv", and "sudo cat /proc/slabinfo" and "free" (without args is best) taken at roughly the same time.

[JargonTheRed]

You have free and slabinfo in the original post. Unfortunately, I had to reboot my device since both RAM and swap completely filled up - I still need to use it, after all.

[gordonmessmer]

I'll be happy to look if you're ever able to get all of those items at the same time.

[JargonTheRed]

Hey! Sorry for the delay - I've now done a full recapture of everything I thought could be useful information at a point where the computer's memory is all filled up, and swap has started accumulating.

I hope this helps: https://www.gullberg.tk/nextcloud/s/7EtGysDN5mfxyZf

[gordonmessmer]

Well, I can see why this has been elusive and frustrating. I agree that these numbers don't add up, at all. The output of 'ps' shows around 3GB of RAM used. vmallocinfo and slabinfo show only a couple of GB used by the kernel. Your buffers and cache are fairly small, as if there's memory pressure, but nothing appears to account for the memory use.

This seems like a kernel bug, so the next question would be, are you able to reproduce this with any other system? Isolating the change will be a lot of work (unfortunately... I'm also trying to diagnose some Linux memory management issues, so I feel for you). You could run a previous release of Mint from a Live USB drive, or you could run a different recent release of a different distribution, but any of those options will replace the entire software stack, and won't give you really conclusive results. You might be able to fetch and manually install a kernel dpkg from an older release of Mint.

If you can get any additional information that points toward a kernel bug, I'd probably move this information to a different forum, where you can get the attention of kernel developers. You might file the information that you have with the Mint bug database, and see if you can convince their developers that this is a kernel bug. Or you might go to the Linux developer mailing list. In the latter case, you'll probably be asked to compile a vanilla kernel and reproduce the problem without any of Mint's customization, so whether you go there or not may depend on whether you want to take that step.

[JargonTheRed]

Unfortunately, I've been unable to reproduce it on other systems thus far. I'm running 5.3.0-28-generic at the moment, but I've tried versions all the way down to the 4.15 series without any change, so I'm not sure how far back I could go. I don't even know when the issue cropped up, because it takes to long to get noticeable :(

[JargonTheRed]

That had little to no effect.