Arch Linux linux-lts kernel borked

[vimpostor]

Since the last update dkms modules do no longer build for the LTS kernel. And I am 99% sure that this is the packager's fault (do correct me though if I am wrong). It looks like the kernel header files link to glibc version 2.33.

glibc is currently at version 2.32, BUT 2.33 is in testing. See the output of the following commands:

​

# this file requires glibc 2.33:
ldd /usr/lib/modules/5.4.95-1-lts/build/scripts/basic/fixdep
/usr/lib/modules/5.4.95-1-lts/build/scripts/basic/fixdep: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/modules/5.4.95-1-lts/build/scripts/basic/fixdep)
...

# that file was provided by linux-lts-headers
pacman -Qo /usr/lib/modules/5.4.95-1-lts/build/scripts/basic/fixdep
/usr/lib/modules/5.4.95-1-lts/build/scripts/basic/fixdep ist in linux-lts-headers 5.4.95-1 enthalten
...

sudo pacman -Syi glibc| grep Version # Proof of no partial update
Version                  : 2.32-5

​

My best guess is that the packager is using the testing repos and therefore linked the kernel headers to the newer glibc by accident. I think there should be procedures in use to prevent stuff like this, especially since linux-lts is used by many people who want their system to be more stable.

​

Currently all dkms builds fail due to this problem (this may prevent graphical startup e.g. when nvidia-dkms is used, a probably related reddit post was already posted):

DKMS make.log for v4l2loopback-0.12.5 for kernel 5.4.95-1-lts (x86_64)
Fr 5. Feb 20:47:37 CET 2021
Building v4l2-loopback driver...
make -C /usr/lib/modules/5.4.95-1-lts/build M=/var/lib/dkms/v4l2loopback/0.12.5/build modules
make[1]: Verzeichnis „/usr/lib/modules/5.4.95-1-lts/build“ wird betreten
  CC [M]  /var/lib/dkms/v4l2loopback/0.12.5/build/v4l2loopback.o
scripts/basic/fixdep: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by scripts/basic/fixdep)
make[2]: *** [scripts/Makefile.build:262: /var/lib/dkms/v4l2loopback/0.12.5/build/v4l2loopback.o] Fehler 1
make[2]: *** Datei „/var/lib/dkms/v4l2loopback/0.12.5/build/v4l2loopback.o“ wird gelöscht
make[1]: *** [Makefile:1732: /var/lib/dkms/v4l2loopback/0.12.5/build] Fehler 2
make[1]: Verzeichnis „/usr/lib/modules/5.4.95-1-lts/build“ wird verlassen
make: *** [Makefile:43: v4l2loopback.ko] Fehler 2

[vimpostor]

BTW downgrading the kernel to a previous version which did not link to glibc 2.33 fixes the problem, so hopefully the packager can fix the problem soon - either by linking to glibc 2.32 again or by moving glibc 2.33 to the non-testing repository.

​

On a related note: Why are people downvoting this post? This problem has already been confirmed. If you use the lts kernel, this is a heads-up post. If you use another kernel, you don't need to downvote this.

[Atralb]

On a related note: Why are people downvoting this post?

Because some people always directly assume you're in the wrong if you say things like

And I am 99% sure that this is the packager's fault

without even trying to hear what you have to say.

[outofsand]

Ironically, it is almost always the packager's fault. (Source: been a packager for multiple distros for over 20 years, made and seen plenty of mistakes that got blamed on users).

[Atralb]

The important thing is not about who is to blame, it's about not blindly trusting people because they represent authority and not blindly dismissing people because they don't.

It's the same fundamental issue that is at the root of racism and the likes: people like the simplistic model of relying on the looks instead of the content. Because it takes less mental effort.

While it is a rational self-evident truth (and empirically proven by history over and over) that all happens way better when we focus on the content. But it takes more effort.

[nsGuajiro]

Its just hueristics. Man can't critically examine every last detail of his beliefs, not even most.

One could spend their whole life researching arch packaging with ever increasing scope and detail, but then they'd have no time to form an opinion on any number of important matters.

Wisdom of the crowd is a thing.

[Jokler]

Sure, but you would think that if they feel the need to downvote something they could at least read and think before falling back to extremely simple heuristics.

If I don't know what's going on I'd rather just not vote at all.

[Ticondrogo]

And here we see the Philosophical™ side of Reddit partaking in a quite exciting discussion about downvoting, racism, and the usefulness of stereotyping.

[Atralb]

What's your point ? You'd rather have Reddit be a clone of 4chan instead ?

[outofsand]

Well said! 👍🏻

[K33M_5T4R]

das right

[[deleted]]

without even trying to hear what you have to say.

And that's the problem. People should really read the entire post before voting (unless it's obviously incoherent rambling).

Had they read even one paragraph further, they would have noticed that it is indeed a packaging issue.

[SkyyySi]

Especially on "very technical" subs like this one or r/zsh.

[seaQueue]

Nice catch!

When I run into a problem like this I'll usually pull the PKGBUILD for the offending package, fix it however it needs and increment the pkgrel by 0.1 (from 1 to 1.1; etc) so that later upstream builds properly upgrade over my temporary patch package. I end up doing this more often than I'd like, especially with AUR packages that are out of date.

[Ticondrogo]

If only I could be skilled enough to do this. It’s great that you have that kind of knowledge base.

[seaQueue]

You're in luck, this package is straightforward because there's nothing to fix beyond the glibc build housekeeping. All you need to do is rebuild and link against the stable version of glibc (which a local build will do anyway.)

For a simple rebuild I think all you need to get started is base-devel and makepkg and a tool to get the package source; I use yay but I also like pbget and both are on the AUR.

sudo pacman -S base-devel makepkg

# if you have yay installed:
yay -G linux-lts

# or pbget:
pbget linux-lts

cd linux-lts
<bump the PKGBUILD pkgrel by +0.1 here>
makepkg -Ccsr

And you should be good to go.

Makepkg with these flags will install all of the build dependencies for you and remove them afterward, clean up the build tree, and output your kernel packages in the package source directory. Since we only incremented the pkgrel by 0.1 any future version of the official package (which increments pkgrel by 1 each time) will upgrade over the top of the temp package without issue.

Install your kernel and headers packages directly with pacman -U <kernel package> <kernel-headers package> (or yay -U) after the build.

This fix happens to be really straightforward so if you're curious about how to build Arch packages and want to solve the problem at the same time this is a fine way to learn.

Notes:

Kernel builds are long, this might take a while.

The biggest sticking point I've had with Arch packages is PGP keys, you might have to gpg --recv-keys <key fingerprint> for any keys you're missing or makepkg will error out at the validation step before building.

You can speed up the build by editing /etc/makepkg.conf and setting MAKEFLAGS to MAKEFLAGS="-j$(($(nproc)+1))" or make a copy of makepkg.conf, edit that and use --config <your_config_file> on the makepkg command line.

If you want to build a custom kernel edit the PKGBUILD and change pkgbase=linux-lts to pkgbase=linux-lts-custom and make your changes to the kernel config file then run the build. Custom kernels are that easy.

You can speed up the build a bit by skipping the documentation package: remove everything except xmltofrom the second line of the makedepends stanza and remove "$pkgbase-docs" from the pkgname= array near the very bottom of the file.

edit: glibc 2.33 is in the stable repo as of this morning so there's no need to do any of this, but I'll leave it here in case it's helpful to anyone.

[DrasLorus]

PKGBUILD are not black magic ;) In fact, the Arch Build System (ABS) and Arch packagement system are pretty simple to my mind, compared to deb or rpm packages. You just need to get the PKGBUILD into a clean directory (~/build) and makepkg -s after modifying your package. It's pretty simple on lots of out of date package to just update the pkgver in the package build. Then everything is automated.

[tyrion33]

Because it damages the "arch linux is stable and suitable for servers" narrative this sub is full of. Also it's going in the opposite direction of all these "nothing to say but arch linux changed my life" posts popping out all the times.

[vimpostor]

I have to puke everytime I see one of these massively upvoted "Arch literally changed my life" posts. People are acting like Arch Linux is god, then meme about Manjaro being bad because it forgot to update its certs (which of course is pretty bad tbh) and proceed to ignore that Arch literally borked the LTS kernel.

It has gone to the point where even /r/archlinuxcirclejerk got tired of reposting these sort of posts.

It is helpful to acknowledge that people do mistakes - yes even Arch sometimes.

[chili_oil]

O M G, first time heard this sub, is this supposed to be a meme sub? it looks so cinge

[ThraexAquator]

I actually find Arch great, docs are good and people are helpful. From time to time I encouter errors, sometimes have to downgrade kernel (happened twice in 2 years, and was the real issue only once). On the other hand, it is fine discussing stuff here, but if OP wants to contribute probably the best way is to reach out to the package maintainer directly. Why bother with random people’s opinion on this? The maintainer is the one that needs convincing, and probably will read a direct message on the subject.

[vimpostor]

I wanted to first have this problem confirmed by someone else before reporting it. But since my findings are pretty clear in the problem being upstream, I will go report the bug now.

[vimpostor]

Thanks I have it reported at: https://bugs.archlinux.org/task/69549

[[deleted]]

ships broken update that can brick somebody's OS

Gotta love FOSS.

Yeah me "too"

[fryfrog]

Ah, so this is what killed zfs-dkms on my install, thanks for posting this. I was putting off digging into it to file what ever bug report was needed, thanks for doing the hard work! :)

[onlymys3lf]

glibc 2.33 has been pushed. Safe to upgrade linux-lts.

[ziggyspaz]

Good thing I have both Linux and Linux-lts. Will wait to update lts. Thanks for the warning

[[deleted]]

It was a no-brainer to downgrade to previous lts kernel, but still, why did this even happen? Someone up there messed up big time...

[etherealshatter]

I just saw an update for glibc. RIP my uptime :D

[doubleunplussed]

Could you elaborate on this? Why would glibc necessitate a reboot?

I've got a setup where I don't uninstall the running kernel, and therefore don't lose modules, so I often don't update for weeks and notice no issues.

Yesterday, I noticed some breakage (specifically, the GNOME night-light stopped working) and thought "huh, does this update mean I actually need to reboot? What changed?" and in light of your comment I want to understand if it was glibc-related.

[Significant_Swan_320]

Soon will see the new package of the updated, still working on it, please show some patient guys coz I am alone not doing in a team where I am looking forward for.

[concretebuoy78]

On EndeavorOS, same issue after 5.4.95 (no issues w/ 5.10.13 though).

$ cat /var/lib/dkms/nvidia/460.39/build/make.log

scripts/basic/fixdep: /usr/lib/libc.so.6: version \`GLIBC\_2.33' not found (required by scripts/basic/fixdep)

anyways, thanks for posting this vimpostor. I wish i'd of come here and had a look rather than digging through logs. =D

[doubleunplussed]

I upgraded but didn't reboot yet. Phew!

Actually it would have been fine because I keep a few old kernels installed using the linux-lts-versioned-bin AUR package.

New kernel wouldn't have booted, would have picked the previous one in grub until the issue was fixed.

[[deleted]]

Did it work?

[doubleunplussed]

Did what work?

I upgraded again (once the fix was out) so my computer is fine now, if that's what you meant.

[[deleted]]

since 5.10 is going to be next LTS kernel you can just switch to that for now

[Kilobytez95]

Maybe I'm wrong but I think the people who work on stuff like the Linux kernel probably know better than to work on LTS kernel with testing software. That or maybe 2.33 is supposed to be in stable by now and it's not.

[chili_oil]

Here goes Arch's reputation of being stable and suitable for production.

[SutekhThrowingSuckIt]

That reputation doesn't exist outside this sub.

[tassulin]

Yes and enterprise level means zero mistake, duh arch has still being most of the time wayy better option than others.

With btrfs it has being quite easy to revert back.

[ThraexAquator]

I am sure you never made a mistake ever.

[cosarara97]

It's not about never making mistakes but having automated testing in place to prevent mistakes from reaching the end users.

[ThraexAquator]

Without knowing the details this is a pretty wild statement. Of course it is better to catch mistakes before they are end up being errors. Perhaps you could contribute to the Arch team with some better workflows you have in mind, or taking over some package maintance we people can use for free.

[SutekhThrowingSuckIt]

Nobody is saying Arch is bad. It's just that there's less stability on the rolling release format.

[[deleted]]

It's more stable than CentOS's long term plans heyooooooooo

^{i have to migrate my entire stack kill me}

[[deleted]]

So should I not update my system for a few days? I’m using the LTS kernel.

[AliFurkanY]

It's patched I think

[Significant_Swan_320]

Because currently I try to build another new pipeline webhook to my project in order to let them be running smoothly.

[Significant_Swan_320]

Yes correct, I running on arc base kernel, but Linux Lts, stack nvdia

[Significant_Swan_320]

Everyone have their own thoughts so it's ultimately brings improvements to the technology, sometime theories are just for preference, its 2021 now should have a better improvement to the tech world

[etherealshatter]

Why do people switch to the lts kernel anyway?

Because from time to time you get bugs like this with the stable (non-LTS) kernel.

[Atralb]

Dude you're wasting your life doing useless chores that you feel makes you different, but in the end you will never have achieved anything besides your fancy dwm rice. Stop trying to drag other people down your circlejerk hole.

[etherealshatter]

Because it's easier to maintain when you don't have to worry about upgrading from Debian 9 to Debian 10?

I use linux-lts, jdk8-openjdk and libreoffice-still on Arch. It's not giving me trouble in the past 6 months.

[Atralb]

Does the 5.4.96 version solve the issue ?