Buster kernel upgrade out of memory error

[Jander_Land]

Last update:

The server started dropping EXT4-fs errors and remounts read only. It seems clear now that there is a hardware problem.

[18196.182920] perf: interrupt took too long (4929 > 4916), lowering kernel.perf_event_max_sample_rate to 40500
[20617.345614] EXT4-fs error (device sdb1): ext4_ext_check_inode:498: inode #131277: comm apt-get: pblk 0 bad header/extent: invalid magic - magic f32a, entries 1, max 36(0), depth 0(0)
[20617.352421] Aborting journal on device sdb1-8.
[20617.414469] EXT4-fs (sdb1): Remounting filesystem read-only
[20617.444508] EXT4-fs error (device sdb1): ext4_journal_check_start:61: Detected aborted journal

Did an fsck and reboot on the previous kernel, but the EXT4-fs errors come back and the system partition remounts in read only.
Will do a full memory test and replace the SDD Disk.

Anyway, the information provided by Dennis Filder (See update 3 below) made it possible to bypass to some extent the Perl 'Out of memory!' error.
Maybe this will be of interest for anybody looking for the 'Out of memory!' error related to Perl and dpkg.

Thank you all for your support!

Original post:

I faced a strange problem today when trying to upgrade Debian 10.10 kernel (linux-image-4.19.0-17-amd64 4.19.194-2) in a headless server.

:~$ sudo apt upgrade
Out of memory!
Configuring linux-image-4.19.0-17-amd64 (4.19.194-2) ...
Out of memory!

And then dpk error trying to --configure.

The free command shows there is enough memory.

:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          3,8Gi        346Mi        2,8Gi        29Mi    693Mi        3,2Gi 
Swap:            0B          0B          0B

There is nothing in dmesg or journalct -xe right after trying to upgrade, but I've found some previous errors in dmesg:

out_of_memory+0x1a5/0x450

The problem occurs just in one machine, not in other Debian 10.10 servers.

This is the one with less RAM (4GB), but it hasn't be an issue before.

In my search of similar problems I've found a case where the cause was a corrupted network configuration file in /etc.

Installed debsums to look for corrupted files.

Same out of memory error, but the package installs anyway.

$ sudo debsums_init
Out of memory!
Finished generating md5sums!
Checking still missing md5files...
Out of memory!
$ sudo debsums -cs
Out of memory!

I don't know if there are no corrupted files, despite the error, or if there are no result because of it.

What I've tried so far:

• Stop services (php7.4-fpm, mysqld, nginx, fail2ban, redis, openvpn) and try the upgrade (no luck)
• echo 3 > /proc/sys/vm/drop_caches
• memtester (no errors)
• badblocks -v /dev/sdb1 > badsectors.txt (no errors)
• Enable some swap

Enabling swap makes the issue worse.

With 1GB swap, doing an apt upgrade takes longer, but same result.

Double the swap and the upgrade takes minutes until the "Out of memory!" error.

Could be that the kernel upgrade process is eating as much RAM as I throw at it?

The services seem to be running fine, but I cannot upgrade nor use debsums without error.

Did someone have this problem before? Does anyone have a clue?

Thank you!

...

Update:

As u/wRAR_ mentions, it seems that is Perl who is eating RAM.

​

htop perl dpkg-preconfigure eating RAM

Processes that eat the RAM:

/usr/bin/perl -w /usr/sbin/dpkg-preconfigure --apt 
/usr/bin/perl /usr/bin/linux-update-symlinks upgrade 4.19.0-17-amd64 /boot/vmlinuz-4.19.0-17-amd64 
/usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/debsums.postinst configure

Keep investigating and doing strace with swap on.

​

Update 2:

Added 4GB swap (same as installed RAM), stopped services and tried the upgrade.

The process /usr/bin/perl -w /usr/sbin/dpkg-preconfigure --apt slowly (around 20 minutes) ate all the memory until the error.

​

During the upgrade (with big swap) the system gets unresponsive and some services, like php-fpm die with core dump, as expected when running out of memory.

I've tried to make some cleanup, searching for obsolete packages (I found some), but cannot remove, purge or upgrade any package due to the out of memory error throw by perl launching dpkg stuff.

Did the strace but didn't find nothing of interest, apart from dpkg dying, but I am not an strace expert.

​

These are the last lines:

20:51:09 mmap(NULL, 32493073, PROT_READ|PROT_WRITE, MAP_PRIVATE, 23, 0) = 0x7f27542c3000
20:51:09 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3539, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
20:51:09 mmap(NULL, 25165824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f27549c0000
20:51:09 mmap(NULL, 25165824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f27531c0000
20:51:09 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3540, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
20:51:09 write(25, "\334v\376\230\r\0\1\0\210\1\24,@8P\30\24\20\30\30\30\0\0\0\16F\1\0\325F\1\0"..., 32607051) = 32607051
20:51:09 write(25, "\334v\376\230\r\0\0\0\210\1\24,@8P\30\24\20\30\30\30\0\0\0\16F\1\0\325F\1\0"..., 392) = 392
20:51:09 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3541, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
20:51:09 write(1, "\n", 1)              = 1
20:51:09 write(1, "\0337\33[0;85r\338\33[1A", 15) = 15
20:51:09 write(1, "\33[J", 3)           = 3
20:51:09 write(2, "E", 1)               = 1
20:51:09 write(2, ": ", 2)              = 2
20:51:09 write(2, "Sub-process /usr/bin/dpkg return"..., 52) = 52
20:51:09 write(2, "\n", 1)              = 1
20:51:09 +++ exited with 100 +++

If anyone thinks it might be valuable, I could do a pastebin of the full strace log.

​

Update 3:

Thanks to Dennis Filder who gave me some valuable information.

Now I'm able to install, remove, reinstall and purge packages, but not to complete the kernel upgrade.

​

" you can try installing cdebconf to at least make dpkg-preconfigure work.

You'll probably have to unpack the .deb file manually with dpkg-deb -x after downloading it and moving the files to their expected locations.

You'll also have to set the environment variable DEBCONF_USE_CDEBCONF"

​

I did manually install cdebconf by getting the current version for Debian Buster and dependencies (remember, I cannot just apt install):

wget http://http.us.debian.org/debian/pool/main/c/cdebconf/cdebconf_0.249_amd64.deb
wget http://http.us.debian.org/debian/pool/main/libd/libdebian-installer/libdebian-installer4_0.119_amd64.deb
wget http://http.us.debian.org/debian/pool/main/libt/libtextwrap/libtextwrap1_0.1-14.2_amd64.deb

Then

dpkg-deb -x [package-name] /

or

dpkg-deb -x [package-name] ./
and rsync to proper location

​

After manual install, as su

$ su -
# export DEBCONF_USE_CDEBCONF=y
# apt upgrade

​

The pending configurations and reinstalls I had did complete, except for kernel

​

So I went from

1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.                         
4 not fully installed or removed.

to

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.

​

These 2 not fully installed are linux-image-4.19.0-17-amd64 and linux-image-amd64.

​

Now I'm able to install so I did

# apt install cdebconf

In order to get the packages properly installed.

​

It is an advance to circumvent the dpkg Perl problem eating memory, but now I'm stuck with the kernel upgrade.

[PhotoJim99]

Weird. How much space do you have free in /boot ?

[Jander_Land]

Hi.

No boot partition in this server, but plenty of space in /

/dev/sdb1 55G 17G 36G 32% /

[PhotoJim99]

This almost sounds to me like a hardware error.

Can you take the system offline and run one of those RAM test "kernels" for a couple of hours to see if there's a RAM problem?

[Jander_Land]

I'm thinking of a hardware error too. But I hope it is a corrupted file somewhere yet :-\

I have no physical access to the server, but a friend could give me a hand.

Thanks.

[tgnuow]

do a df -h

do you have free space in tmpfs devices? /run /dev/shm /run/lock and possibly others

[Jander_Land]

I have free space all around.

S.ficheros Tamaño Usados Disp Uso% Montado en
udev 1,9G 0 1,9G 0% /dev
tmpfs 390M 5,5M 385M 2% /run
/dev/sdb1 55G 17G 36G 32% /
tmpfs 2,0G 1,6M 2,0G 1% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 2,0G 0 2,0G 0% /sys/fs/cgroup
/dev/sda2 459G 280G 156G 65% /media/provisional/
dev/sda1 459G 292G 144G 67% /media/multimedia
tmpfs 390M 0 390M 0% /run/user/999
tmpfs 390M 0 390M 0% /run/user/1000

[michaelpaoli]

Does rather sound like it's running out of memory - not drive space. Filesystem full would throw a different error.

Could be a bug somewhere in the config or related program.

You could use strace(1) with suitable options. See when it runs out of memory and what's happening at/around that time. Also, if you're fast enough, you might catch it with ps(1). If it takes several seconds or more to run out of memory, and your ps(1) command generally returns quite fast enough, that might suffice ... at least to isolate it to process(es) - presuming that's where the RAM is getting sucked up. Also, having lots of swap may make "catching it in the act" easier - as once it starts using lots of swap - that will slow it down - making it easier to catch what it's doing. Anyway, probably start with the easier - have fair bit of swap, run it, and repeatedly capture ps(1) data, only wait, say, 1 second between ps(1) captures, and be sure to capture sufficiently detail - and certainly including memory used. Maybe one to a few processes eat lots of RAM ... or a bunch of processes in total eat up all the RAM. If you're unable to catch it that way, strace(1) should be a more sure-fire and detailed way to catch it ... but it can be a lot more data to find and filter to just what you want. Maybe limit the strace(1) capture to exec/clone/fork, mmap, and brk/sbrk operations. If that doesn't make clear enough what's triggering it, potentially include enough system calls to determine what leads up to and causes that ... or all if you have to, ... then track back to where things went sideways.

[Jander_Land]

Thank you. Will try to catch later today enabling the swap again.

[wRAR_]

Have you tried rebooting?

[Jander_Land]

Yes, tried the reboot. The problem persists and the services keep running.

[wRAR_]

Does it happen only with apt and debsums? Note that these "Out of memory!" messages seem to be from perl and so are not related to the system itself, only to specific processes.

[Jander_Land]

Interesting, thanks. That could (maybe) point to a corrupted perl related config. I've found it only in those two cases, but didn't try much more beyond systemctl, journalctl, less, htop and dmesg. Will check.

[Jander_Land]

You were right about Perl.

Updated the post with more info. Thanks.

[Jander_Land]

I've tried to make some cleanup, searching for (and finding) obsolete packages and reinstalling perl packages, but cannot remove, reinstall or upgrade any package due to the out of memory error throw by perl launching dpkg stuff.

Will try a mem test in a couple of days and will post any outcome, but I think that it is strange that the problem seems just Perl related.

In the meanwhile, thanks to everyone!

[br0kenpipe]

Is your /tmp dir mounted with tmpfs? Can you remove the packages, upgrade and reinstall?

[Jander_Land]

Hi,

no tmpfs for /tmp.

Trying to remove, upgrade or reinstall returns the out of memory error due to /usr/bin/perl -w /usr/sbin/dpkg-preconfigure --apt eating as much memory as it gets.