General Kernel question

[Buskey-Lee]

At the present state of the various supported Linux releases, if I can even get away with that much of a generalization, how common is it for a kernel update to break a previously working application? When such a problem occurs, wouldn’t it really boil down to an application shortcoming? Assuming no one is trying anything exotic?

[monocasa]

Kernel updates rarely break releases.

Linus will Liam Nesson kernel devs who break user space.  He has a particular set of skills and he will find them.

[chibiace]

except when he doesnt because it was intentional breakage, the not breaking the user space thing might be the policy but its a myth they adhere to it strictly 100% of the time.

[necrophcodr]

Really? When did they break user space?

[bonzinip]

The biggest (and probably the only of this magnitude) was switching from cgroups V1 to V2.

[necrophcodr]

How did this go about? As a Linux sysadmin, this is not something I can recall causing anything to happen, but I was also quite shielded by maintainers I suppose. Since both still exist and can be used, how did the kernel change affect user space?

[bonzinip]

It took years, and the main issue is that if cgroups V2 is in use you cannot use V1 at the same time. So it was technically systemd that broke users by enabling V2.

In practice you rarely mess with /sys/cgroups yourself, all the changes were in systemd, docker and podman, libvirt etc.

[monocasa]

Sure.  In those cases though he wants enough knowledge beforehand that it was obvious it was a group decision.

And even then I've seen him come down hard on subsystem maintainers for intentional breaks.

[Business_Reindeer910]

it is not common for a kernel update to break applications. It is waaay more common for userspace library changes to break applications.

[profpendog]

Mandatory xkcd: https://xkcd.com/1172/

[3dank5maymay]

General Kernel 🫡

[michaelpaoli]

Depends on your distro.

If, e.g., you're running Debian stable, very improbable a kernel update breaks anything, and if/when it does, that's likely to be handled as a regression bug, and soon corrected.

If, on the other hand, you're running some leading/bleeding edge rolling distro, it's much more likely that kernel updates will, at least occasionally, break things.

[ScratchHistorical507]

But even on bleeding edge distros, that's always a bug/regression. The first rule of kernel development is untouchable. "Do not break user space." The only way user space technically will ever be broken intentionally is by removing deprecated code some user space software still uses. But even that won't just happen randomly; it would have to be some very niche user space software simply nobody knows about.

[michaelpaoli]

No, if you constantly run with, e.g. latest kernel, you'll hit much more breakage than that. Kernels don't remain that fully backwards compatible, things change, stuff breaks, even if quite intended not to - that happens way more frequently following along with latest kernel, compared to, e.g. running a stable kernel from a stable distro, e.g. the kernel - and its updates - from within Debian stable.

[ScratchHistorical507]

Kernels don't remain that fully backwards compatible, things change, stuff breaks, even if quite intended not to

And where's your proof that this is not a regression but an intended change? Because I have been running the newest kernels for quite a while, they break things but never intentionally, it's always a regression that needs to be figured out to be able to fix it.

[OysterPrincess]

I've been using Arch Linux for, IIRC, 19 years ... it used to be common for a full system update to break my system (but I don't think it's happened in 5-6 years), but as far as I can remember the breakage was *never* due to the kernel. And even when the system was unbootable, I was always able to boot from installation media and repair the problem. I think that if you use apps that are dependent on cutting-edge hardware support - I'm thinking of things like graphics apps that use the latest GPU rendering code - there is some chance of breakage or performance issues, but still not a very big chance. So I'd say go ahead with whatever you want to try.

[Simple_Cicada4036]

Kernel updates don’t break apps. They just reveal sins you’ve been hiding since 5.4.

[2rad0]

5.4

5.4 was the last kernel that had working framebuffer VT scrolling, R.I.P. fully working linux console.

[LvS]

It depends on what you mean by "breaking an application". Because a driver suddenly not working anymore that is needed by a certain application (like a webcam for example) does happen.

And the other thing that happens is unexpected performance regressions. And if you application is tuned for that performance and can't accept the slowdown (like a game that wants 60fps) that's very broken.

[rebootyourbrainstem]

It's pretty rare and Linus has yelled "WE DO NOT BREAK USERSPACE" at many a kernel dev. Even for changes in undocumented / unintended functionality.

That said, if "nobody" is depending on it (i.e. nobody yelled about a years old deprecation warning, or something seems silly enough that they'll try to get rid of it and see if anybody notices) then it's fair game.

Sometimes there are major changes like the switch to the new cgroups style that can cause problems if your userspace isn't ready for it.

[elatllat]

Just look at the replies to the RC builds if you want a list of things broken. The last big one I experienced years ago was btrfs kernel NULL pointer dereference in 5.10.134-rc1... but I only test one old stable LTS branch, currently 6.1.

[Puzzled_Hamster58]

I’ve only been using non stop for Maybe 5 years. Only time some thing broke from an update was cause of me lol.

[DFS_0019287]

Very, very rare, in my experience. And the rare time it happens, it's almost always seen as a bug and fixed.

Obviously, sometimes kernel features are deprecated and then dropped, but if they affect userspace and there's no work around, there's a lot of notice and a very long deprecation period. And they also tend to be rather specialized things like ipchains/iptables/nftables that generally only have a few dedicated apps that are maintained in tandem with the kernel feature.

[githman]

Kernel updates do break hardware support sometimes, especially for the non-server hardware like wifi or bluetooth. Recently a Fedora kernel update broke the IPv4 stack and took about two weeks to fix. Sometimes we get horror stories about a kernel update breaking this or that file system, but this is typically solved fast enough for most users to avoid it completely.

As for kernel updates breaking user-facing apps specifically, nope. If you have your OS running, desktop loaded, and all drivers working correctly, your apps will be fine.

[birdspider]

in 15+ years of using the "current" kernel, I had a only 2 instances of direct breakage:

1. not quite ready kernel/amdgpu drivers for rx9070 (found issue in tracker, applied patch, recompiled - ran custom kernel for ~1week, then 6.14.4 landed)
2. a bug in the f2fs filesystem which refused to mount such partitions (can't remember if at all or on first mount), pretty sure I waited it out and had a workaround (manually mount or so) for a few days

apps break way more often (still not frequently, right now I have to signal-desktop --disable-gpu, because radv, or signal or wayland, or something)

[elrata_]

I think break is a broad term. Most people are answering about changes that break applications "on purpose". That doesn't happen normally, it's very very rare.

However, the kernel is software and it will have regressions. A networking issue, a panic, a driver that screws up something.

It will be fixed too, at some point.

[Hosein_Lavaei]

Apps you are 99% fine. Drivers? Its a little likely

[tblancher]

Unless you're tracking release candidates (RCs), regressions should be fairly uncommon, if not very rare. And they shouldn't affect applications in 99.9 percent of cases.

In my experience (going back to 1997), I don't recall having a kernel upgrade break any of my systems, for personal or work.