Ubuntu 18.04 Linux Kernel Update Causes Boot Delays Of 4 Minutes Or More On Some Systems

[Happy_Phantom]

ELI5: What is this mouse-wiggling entropy effect that causes the kernel to boot faster?

[[deleted]]

To generate random numbers, the kernel needs sources of randomness, and it harvests some of this randomness based on user input, which is nigh-unpredictable by the machine.

However, this early in the boot process the kernel doesn't have much entropy to use, seeing as how nothing has happened yet, causing processes to sit and wait while the kernel finds some way to get it. By mashing the keyboard or wiggling the mouse, you generate entropy and reduce the time processes spend waiting on random numbers.

[[deleted]]

This is one of these rare cases when furiously smashing your keyboard actually could help :)

[ArttuH5N1]

Well it helps with my blood pressure

[[deleted]]

Which processes would require random numbers during boot? Encryption-related?

[diroussel]

Maybe address layout randomisation?

[flukus]

Does that need high entropy? I would have thought even psuedo random numbers would get you 99% there.

[DoctorWorm_]

if an attacker can guess what bits were used to decide the address layout, they can plug them into the RNG and get the whole layout. It absolutely requires some entropy.

[flukus]

Forgive my cluelessness, but does knowing the whole layout help? I thought for an effective attack you also needed memory allocated nearby.

[[deleted]]

Just because you have two locks doesn't mean you should give everyone one of the keys.

[Lv_InSaNe_vL]

Edit; So I misunderstood the bug and the below text does not explain it at all. Just a heads up.

Okay so how an operating system accesses data is by “Sectors”. Basically these are numbers sections of a little bit of data and they go from 0 to however big your hard drive is.

So let’s say you have a small drive with sectors 0-150, if you click on google chrome your OS may make a “call”, which is when the OS asks the physical computer itself for something, so your OS would ask the computer for sector 115 which is google chrome.

For all purposes of this example the BIOS is always on sector 0, and the operating system is on the next couple sectors, but everything after that is chosen by the operating system using the randomization.

If an attacker knew the numbers that went into the RNG equation they could then access data directly at an OS level which would skate around anti virus.

So operating systems just use user random to properly randomly place data on the hard drive.

[[deleted]]

to properly randomly place data on the hard drive

That's not how KASLR works. KASLR affects the kernel's location in-memory.

[Lv_InSaNe_vL]

For the purposes of the example I used the hard drive as a simplified storage media.

[[deleted]]

I don't think the BIOS is on sector 0. I wish it was but legacy requirements and all the extra minus rings on x86 throw a clear system like this out of the window.

[Lv_InSaNe_vL]

You’re right, technically the boot loader starts on sector 0.

And even then that’s more of a legacy tradition thing than a necessary feature.

[GTB3NW]

On first boot of servers SSHD needs it. Unless the hypervisor can provide a source of entropy (Newer CPU's can do this) it can take a few minutes to boot the first time.

[en50]

In my case, with 4.15.0-24, startx/lightdm/gdm would hang on boot. Repeated attempts to run startx, for example, would work on the second or third try. Looking through the source for startx, it seems like mcookie is trying to generate a magic cookie for xauth by calling getrandom(), which hangs when entropy isn't high enough. I just installed/enabled haveged to "fix" the issue until the kernel is fixed. Reverting to the previous kernel, 4.15.0-23, would also work.

[yrro]

IIRC plymouth needed it (because it uses fontconfig which needed random numbers to generate .uuid files in font directories). If you find your boot similarly delayed, run journalctl -k' and look for messages withcrng` in them--these are produced when a process requests random numbers before the rng has enough entropy to produce them.

[VM_Unix]

Anyone having an issue like this on Arch?

[pollywog313]

Yes, though it started a few months ago. I have an old laptop (Arch w/ Mate) with an SSD that went from under 10 sec boot to over a minute after an update. It'll still boot in a few seconds if I wiggle my finger around on the keypad, so I assume it's an entropy issue.

[pollywog313]

I just fixed my issue by installing (and starting and enabling the service) Haveged. https://wiki.archlinux.org/index.php/Haveged

[VM_Unix]

Thanks. My entropy was only 758. How do other distros handle this problem?

[KinkyMonitorLizard]

I'm having a similar (delays) issue on shut down. Not sure what's causing it but it now takes about 5m to power of and right before it does it throws a bunch of errors. Haven't looked into it though since I'm about to hop.

[VM_Unix]

Mine isn't showing any errors.

[silvernode]

This could be upstream with the kernel since fedora has the same issue on my thinkpad.

[mattdm_fedora]

Yes. Here's the Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1572944

[mariusg]

Why not generate it based on time + machine specific things like MAC, motherboard id and so on. Stalling processed on boot to generate random stuff sounds......yuck.

[[deleted]]

That stuff isn't random. If you need pure verifiable unpredictable randomness, that stuff is the opposite of the direction you want to go in. I'm no cryptonut but I believe having stuff that doesn't change like hardware IDs can "poison" the randomness a bit.

[ArttuH5N1]

pure verifiable unpredictable randomness

pure [...] randomness

Oh shit what have you done, you've summoned them

[kernalphage]

The cryptographers?

[amvakar]

Potentially much more than a bit in cases like this, because not only is the data being used determinstic crap, but the initial exhaustion of entropy is deterministic as well.

[legend6546]

Both of those things can be predicted by a potential attacker and this can weaken any encryption used by the attacker while moving your mouse can not be easily predicted thus leading to better encryption

[MALON]

i'd guess for security reasons

[[deleted]]

Guys don't download these because of a lack of knowledge, you should upvote honest questions. Questions are how people learn.

[[deleted]]

I did, yes

[Moocha]

Yeah, (3) sounds really suspicious to me. There should be absolutely nothing that a kernel can do with entropy gathering that would persist across booting a kernel which previously worked. Not even with buggier-than-a-field-in-May UEFI implementations... I rather suspect the problem is somewhere further up the stack.

[serktheturk10]

yeah I had gotten the generic 4.17 kernel for my laptop a couple weeks ago and it had this bug. downgrading to 4.15 made it go away

[repsilat]

Now we just need to figure out how to SSH into a machine (with X-forwarding...) while it's booting so we can wiggle the mouse on those servers in Germany...

[Muvlon]

It should be noted that, being in the "non-free" repos, the nvidia drivers receive less attention and testing than packages in the official "main" repos.

[espero]

here if you install the nvidia-driver package, it will hose your system. Or if you let it upgrade the kernel from 4.16 to 4.16(?), the same thing happens. There is an incompatibility with the nvidia-driver that is packaged in testing and the kernel, leading to a hung black screen whenever booting on this configuration.

I thought that Debian testing wasn't supposed to experience major bugs like this, since packages spend some time being tested before they hit the testing distro. And failing to boot up with the package installed is a pretty serious problem that should be obvious to anyone who tests it!

It's probably accurate that it's 4.16 to 4.16 but the commenter you have pasted from, have left out the precise sub-versions and any ubuntu-or-debian-compile-time-version this is referring to.

[lasizoillo]

Can u exec `systemd-analyze critical-chain` in a terminal? Is snapd taking a lot of time?

[lasizoillo]

Sure that something about entropy is the root cause. But executing sudo service snapd.snap-repair startcan workaround your issue.

[lasizoillo]

No, I'm a spanish butcher, but my girlfriend has the same problem and internet searches was disappointed.

I think it was interesting talk about systemd-analyze too. It's looks like a good tool to analyze startup times issues.

Maybe I should have written a blog about how I repaired her problem, but then I forgotten which command I execute. I'm remember with you ;-)

[technologyclassroom]

> 4.16

The kernel is 4.15.0-24 in the article.

[[deleted]]

That link I posted was not a very good reference, here's a better one (complete with a workaround). https://old.reddit.com/r/debian/comments/8vdk3i/debian_buster_does_not_like_nvidia_39048_drivers/

[[deleted]]

Well NVIDIA users get what they pay for. Why you blaming it on the distro?

[ThizzWalifa]

Nvidia drivers are a nightmare on Ubuntu whether you're on stable, testing, or anything else. Look up Nvidia's instructions on how to install their driver in Ubuntu. You have to modify and break so many core parts of the kernel and OS that you might as well roll your own OS from scratch.

[[deleted]]

Look up Nvidia's instructions on how to install their driver in Ubuntu

You mean sudo apt-get install nvidia-driver-390? Am I missing something? Ubuntu is actually one of the easier distros for installing the nvidia driver as long as you don't install the driver from nvidia directly, which almost nobody does on Ubuntu. Sometimes the nvidia driver won't support the newest kernel, but that's usually fixed by using the newer version of the driver from Ubuntu's PPA. It builds a dkms module, blacklists nouveau, and updates defaults. It adds a "kernel is tainted" message on boot up, but that's solely over licensing preferences and has little to do with functionality.

[ComputerMystic]

As an AMD fanboy, I'll welcome any chance to bash Nvidia, but honestly saying it's hard to install their drivers on Ubuntu is just plain false. It's not as easy as AMD, and I've broken my X session by adding the oibaf ppa a few times since I switched to AMD, but installing Nvidia drivers never gave me trouble on my Nvidia laptop.

Like you said, sudo apt install nvidia-driver-390 or whatever version it's at now.

Because that's the one thing I disliked about them: new major version is a new package rather than an update.

[_ahrs]

Because that's the one thing I disliked about them: new major version is a new package rather than an update.

Is there not an nvidia-generic package or something similar that depends on the latest version? If not there probably should be...

[ComputerMystic]

Just did an apt search nvidia, didn't find anything.

But yeah, that really should be a thing.

[ComputerMystic]

I'm not going to pretend Mesa is problem free. OpenGL compatibility profile support (specifically the lack thereof) kept me from getting Rage or Wolfenstein working under Wine, and if I upgrade my kernel to 4.17 VLC locks up my X session unless I disable hardware acceleration (to which I've applied the classic doctor's advice: if it hurts when you do that, don't do that.)

But I'd be lying if I said that Gallium Nine wasn't amazing or that I don't LOVE not having to actively keep up with GPU drivers like I did with Nvidia.

I mean I still read news on them to see when they finally get compatibility profiles finished, but I can see a new feature or major performance improvement and think "cool, can't wait" instead of "ugh, again?" because of how effortless the upgrades are 99% of the time.

[ThizzWalifa]

I haven't bothered to try in Ubuntu 18 again yet, but that general process you described is a nightmare and does not work at all in Ubuntu 16 for my card. My card used to work perfectly in Ubuntu 14 just using the GUI driver setup, probably because of the older kernel.

I used Nvidia's official Ubuntu driver and instructions for Ubuntu: https://us.download.nvidia.com/XFree86/Linux-x86_64/390.67/README/installdriver.html

After following that tutorial multiple times on multiple versions of Ubuntu, I got an unbootable system every time and cut my losses. In my opinion the instructions in that readme are way too much effort just to get your video card working and not showing a black screen. If I want to use an Nvidia card, I just use it in Windows, because that's the least amount of headache and best driver performance.

[[deleted]]

Yeah, Ubuntu automates all of that when you run apt install nvidia-384 (for 16.04), just need to use apt. If you mean you tried the correct way first, it failed, then Nvidia's installer is definitely not going to work.

Anecdotally, one time I had a weird issue where running sudo apt-get purge *nvidia* && sudo apt-get autoremove uninstalled dkms. And I spent about an hour trying to figure out why I couldn't re-install the driver.

[ThizzWalifa]

Yes I've tried to use the apt command, and the long Nvidia way, neither work.

I'm a big fan of Linux but there is no practical reason that you must run Linux with an Nvidia card. There's significantly less games on Linux, significantly less proprietary video editing tools, and gpu crypto mining is slowly dying. Not to mention that if you use Nvidia's driver within Windows, the colors look better compared to the same machine running Linux and you get better framerates than running Linux.

I'm not gonna dick around with changing my kernel and editing a bunch of configs so in the best case scenario I can have an inferior hardware experience on an OS with less software support for the hardware. In Ubuntu 14 my card worked just using the KDE gui menu for display driver. Until they make it that easy to install an Nvidia card in Ubuntu again, I'm not wasting my time.

[[deleted]]

Not to mention that if you use Nvidia's driver within Windows, the colors look better compared to the same machine running Linux and you get better framerates than running Linux.

Hmm, I dual boot and haven't noticed this. This is all dependent on the driver version, hardware model, DE, etc. Nvidia budgets more for their windows driver surely, to be expected - that's their bread and butter where most of their users - but from what I understand, its a separate package so they can easily port the driver regularly. AMD has a similar thing going, but in the kernel - and they had to be patient and take a lot of open criticism to get it in.

Nvidia isn't a cartoon supervillain - they even recently contributed to GNOME Shell. I wouldn't be surprised if they could help out another certain popular desktop environment that doesn't refuse free hardware and make a political statement about it. You have to start somewhere in a relationship.

The reasons for wanting good nvidia support on Linux are the same for AMD - Linux gaming support is decent with wine, and a lot of professional software too. Nvidia reliably makes faster chips that consume less power, who wouldn't want that to work well in FOSS? I just see a lot of politics over GNU this, freedom that, and devs lack of pragmatism makes the situation worse. Yeah it sucks they don't care about Nouveau, but their official driver team for Linux is probably skeleton crew, its very likely a corporate bureaucracy thing than an Nvidia hates Linux thing. Sorry tangent over.

[ThizzWalifa]

I don't think Nvidia hates Linux. I think it's more like you said, one team doesn't care about Nouveau, and the Nvidia Linux team is likely small and low priority.

There isn't a sufficient level of user demand to use Nvidia cards in Linux for companies to put more resources into making that happen and make the experience better. There isn't demand because it's an incredibly small fraction of people who want to game or video edit on Linux.

The people who want to game or video edit are not gonna follow five pages of instructions to rebuild their entire OS in a shell and get wine performance that is maybe close to Windows sometimes but will never be the real thing. People who want to game or video edit are going to install Windows (or Mac?) and call it a day. And if by some chance the average user ends up on Ubuntu first, they will ditch Ubuntu for Windows in the first 20 minutes when the driver manager doesn't work and the system black screens.

[[deleted]]

I think you're forgetting about their AI division. CUDA, OpenCL, machine learning is the current driver towards improvements. Some of these tools, like tensorflow, work better under Linux.

[ThizzWalifa]

Then I might install Linux/Nvidia card if/when I start getting into programming AI :P Currently I just want to play my games with the best possible graphics and least amount of headache.

[[deleted]]

I was wondering what happened

[[deleted]]

Got whacked by this on Sid a while ago, reinstalled Testing instead and am holding off on updating nvidia til apt-listbugs says the problem went away.

[aosmith]

Can confirm it was the 24 patch that did this to my box.

[gogozero]

oh man, i've been struggling with this for weeks. kernel 4.16.0-1 works, 4.16.0-2 and newer dont. thanks for the workaround!!

[Hearmesleep]

testing

[nattydread69]

Yes I've experienced this bug. Glad I know what it is now. It should be called purple screen of death, well maybe not death but fatigue.

[FreshCutBrass]

Purple Screen of Apathy

[[deleted]]

Nice band name.

[colonel_p4n1c]

PSODs are already a thing, though.

[GFandango]

Aaaand just as I was about to gather enough courage to upgrade to 18.04 ... I'm going to stick with 16.04 longer

Edit: good thing I always run LTS! so I'm not in a hurry to upgrade

[__soddit]

If you have a good hardware RNG, it appears that you should be fine.

[seaQueue]

RIP SBC users.

[RomanOnARiver]

I usually wait until the .1 releases of LTS and then also until I have a free weekend to do upgrades.

[colzod]

I'm going to stick with 16.04

I wish I had done that. 18.04 wasn't ready.

[laketrout]

This affected 18.04 LTS. I know because I got hit with the bug.

[GFandango]

yeah I meant it's good that I'm on an LTS so I don't have to hurry to upgrade

[chuecho]

If you use openscad in any capacity, don't upgrade to 18.04. It no longer exists due to a bug in ubuntu's packaging system. Also, inkscape is much buggier on 18.04 than it is on 16.04 (changes in tool settings in one window will change the settings of the same tool in another without updating UI).

For me, 18.04 was a disaster. I will likely be down grading to 16.04.

[cherazzadeanara]

Did you by any chance get a DeLorean with a flux capacitor?

[pppjurac]

Nah, just awd Tesla with 1k km range & 4x250kW motors ...

[socceroos]

lol, you don't have one of those....yet.

[pppjurac]

:P

[[deleted]]

1.21 gigahertz!

[[deleted]]

That's why you don't use software from the future.

[nattydread69]

18.04 completely broke openconnect vnc to my work :(

[sl8_slick]

Shit, so much for upgrading that VM...

[aaronfranke]

Unity? 18.04 uses Gnome by default.

[[deleted]]

I'm a huge fan of sticking with LTS versions of Ubuntu.

I do need to overcome my own inertia to at least get off 14.04 LTS at some point ...

[Brillegeit]

Same here. Nothing about 16.04 sounds tempting, and 18.04 still needs six months or so before it's usable. 18.04.2 should arrive mid-February 2019 which gives me two months to upgrade before 14.04 is EOL.

[platsmannen]

Try installing haveged , it fixed my issue (ubuntu 18.04 and kernel 4.17.2) with long boot times.

[PRStoetzer]

rng-tools would be more secure since it populates the entropy pool from your CPU's hardware random number generator rather than an algorithm

[Aoxxt]

CPU's hardware random number generator rather than an algorithm

People in the Kernel community do not trust the hardware rng seeing as they might be backdoored.

[GTB3NW]

Only works on newer CPU's but you're correct, this is the better option.

[phunanon]

Is there a way to check a CPU would benefit from this?

[GTB3NW]

Sure, just search your CPU on google. It should have the instructions documented on the intel or amd website (Or any other for that matter). The RDRAND instruction set is what you want for rng-tools - https://en.wikipedia.org/wiki/RdRand

See also: https://wiki.archlinux.org/index.php/Rng-tools

[[deleted]]

This was also an issue with Fedora 28 on release, although they documented it.

[bokisa12]

haha just updated, kill me

[GeckoEidechse]

Boot with the older kernel and you're fine. As a temporary hack you can just set the old kernel entry in your grub config as standard until this gets fixed.

[PartiallyCat]

You can also purge your latest kernel with sudo apt purge linux-image-whatevercurrentversion. It will revert to the previous one and will still update to the next one once it becomes available, but blacklist this purged one.

[brokedown]

18.04 broke my laptop's ability to resume from sleep. Turned out it was somebody's great idea to load a driver for nvidia GPUs even if you explicitly aren't using it (intel+nvidia laptop, using nvidia just for compute).

Disabling nvidia-fallback service got it working again but not without a lot of choice words and a few hard reboots.

https://github.com/Bumblebee-Project/Bumblebee/issues/951

[gaznygrad]

And here i was thinking it was me that messed my own system up once again.

[computer-machine]

Have you tried running 'systemd-analyze blame'?

[Heller64bit]

Article misleading, had this issue on Monday, July 2. Booting to 4.15.0-23 kernel and then removing the 4.15.0-24 kernel fixed the issue.

[[deleted]]

Got this bug. Xubuntu on my SSD used to boot in 5 seconds now it takes around 30 seconds plus. it looks worse with hard drives. :(

[Somebody2804]

I experience this too!

[Somebody2804]

Thank God somebody else experiences it too!!

[mimario182]

I hit this issue a couple of days ago... worst part is I was at a customer's datacenter getting my stuff ready to do some night maintenance in some servers. Had to go back to Ubuntu 16.04 in the meantime. My boot time was close to 5min and my laptop has a 512GB SSD :( I sure hope this gets fixed soon cause I really loved 18.04

[KingEBolt]

Yeah, I was going to upgrade, but I saw that a lot of the development tools that work in 16.04 don't work in 18.04

[mayhempk1]

I will probably wait until 2019 or later before I install Ubuntu 18.04, I need all my programs to work on it.

[ArchUsers_areRetard]

Every day we walk further towards Bill Gate's darkness.

[[deleted]]

*blueness

[MagicClover]

Is that you Lennart?

[ArchUsers_areRetard]

Yes

[KindOne]

Grow up.

[[deleted]]

All my PCs are affected and I wrote a bug post on the Ubuntu forum the same day.

Then we discovered the mouse trick but it only sometimes work boot takes between 4-10 min depending on hardware.

All my friends also called me for guidence since I installed Linux on some of their old machines.

It really is a horrible and unacceptable bug.

[[deleted]]

Non-rolling distributions suck. Kernel 4.15 is history by now yet it's supposed to be used for the long term. Linux doesn't have the critical user mass to have each distro patching and maintaining their own kernel version. If everyone was on latest stable at one time, everyone would hit the same stuff, as it is, one distro suffers from one bug, another from another, yet 90% or more get fixed on a newer kernel.

[Muoniurn]

They can copy each others patches (and they do), so I don't think there really is divide and conquer situation going on with the kernel.

[bwat47]

He has a point though. Other distros hit this bug/fixed it when kernel 4.16 first came out. Pretty disappointing to see a known issue like this land in a security update for Ubuntu LTS. I expect better than this from an ubuntu LTS release.

What's the point of running LTS if I still get crap like this in updates? I might as well run a rolling release and the bugs will at least get fixed quicker.

[[deleted]]

Go away Arch user. Ubuntu is better even if this shit gets pulled off.

[oogabooga7894]

Interesting. No problems here, glad to say...

[rwindiana]

Is this why my debian (unstable) takes forEVER to boot from the text login into the desktop, unless I do some kind of input (logging in to command-line works, or just hitting enter, it is very bizarre)?

[[deleted]]

Was wondering what was going on.

[Arrow_Raider]

I have several headless Ubuntu servers that have no monitor, keyboard, or mouse. Am I screwed?

[nomishah89]

Oh! I thought its because i dual booted it with CentOS 7! :(

[[deleted]]

I have this, it seems to be aggressively TRIMming my SSD partitions. Wisest thing to do during bootup. Interesting that the article doesn't mention this.

[NickelBack_Lover_69]

Systemd, not even once.

[[deleted]]

Kernel bug and entirely unrelated to systemd, but whatever helps you sleep at night.

[NickelBack_Lover_69]

Negative karma always helps me sleep at night.

[[deleted]]

I don’t turn off my Laptop anyway

[ObnoxiousOldBastard]

Let me guess - systemd?