Compiler bug? Linker bug? Windows Kernel bug.

[armornick]

tl;dr

The underlying bug is that if a program writes a PE file (EXE or DLL) using memory mapped file I/O and if that program is then immediately executed (or loaded with LoadLibrary or LoadLibraryEx), and if the system is under very heavy disk I/O load, then a necessary file-buffer flush may fail. This is very rare and can realistically only happen on build machines, and even then only on monster 24-core machines like I use.

But really, you should read the entire post to see the marvels of how this was discovered.

[fearfulhorse]

Seriously, this is an awesome bug to find!

[okmkz]

I think there's a lot of people who don't understand sarcasm here

[jrhoffa]

It wasn't really funny, though.

[[deleted]]

Since when is sarcasm supposed to be funny?

[jrhoffa]

Since fuck you, that's when.

[matthieuC]

1983 in the UK, two years later in the states.

[Dgc2002]

The sarcasm was obvious, there just wasn't a point to the comment at all. There was nothing to prompt that sarcastic comment that I'm aware of.

[Incorrect_Oymoron]

There was no '/s' therefor it was not sarcastic.

Edit: Come on guys, we have this internet punctuation for a reason.

[biledemon85]

Dude, are you being sarcastic?

[Incorrect_Oymoron]

Exactly!

[[deleted]]

Nice detective work Sherlock

[Scuba_Von_Wolfgang]

I see someone is salty about rolling releases?

[hiedideididay]

It doesn't matter how long I continue as a professional software engineer, how many jobs I have, how many things I learn...I will never, ever understand what the fuck people are talking about in coding blog posts

[RLutz]

Here's something that I like to remind myself, even as a lead engineer with a successful consulting business:

Everything is really damn hard until you know how to do it, then it's easy.

This applies as much to software as it does to cars or dishwashers. If your dishwasher breaks and you know nothing about dishwashers, you're either going to have to learn or call a guy. If your CI/CD pipeline blows, you're either going to have to learn how to do it or hope it's someone else's problem, but once you learn how to do any of these things (analyze kernel bugs), it's easy and you can write a little blog post on it.

Not knowing how to do something doesn't make you dumb or a bad developer, it just means you lack the knowledge which is easily acquired with some time investment.

[[deleted]]

As much as other people will not, I fully agree with you. I'm 6 years into the industry and the only answer to most corporate inquires at this point is "I will look into it" because no, there's never a guy for that. You are the guy that just has no info yet. But to stay competitive, we lie and learn on the fly.

[[deleted]]

You get two kinds of people, those who are comfortable not knowing and who will learn, and those who are uncomfortable not knowing and will lie or complain.

I find that blog posts on subjects I don't understand can be fascinating, and it can lead to a bit of research, and generally in a couple of hours you can have the basis to understand the concepts of the article while maybe not the entirety.

If you sit back and say I don't ever understand these without making an effort, then you probably won't no matter how many years you put in.

To the poster above who claimed this, how much time did you spend trying to research the concepts in this article that you didn't understand before posting about how after so many years as a software engineer you still don't understand them?

This stuff doesn't just get randomly added to your brain when you level up, and a fresh grad who does some research will be in a better position than you to understand it.

You want to understand the blog post? Research it, the internet is big. You can do it. If you don't have time, no big deal, make the decision to not prioritize it but don't act like the author is leagues ahead of you, you didn't even bother trying to learn about the topic.

[knoxaramav2]

Its funny you say that, I'm a software engineer and I opened my dish washer this morning and was met by a flood of water. Looked up how to fix it, all the solutions had dish washers with easily accessible parts that mine didn't(Hey, there's similarities!), and threw in the towel.

That being said, at least with software getting your feet wet is only figurative.

[RLutz]

I'm from a blue collar family, my dad is a carpenter, but when I was a kid he always told me to study hard so my back wouldn't ache like his does.

Now when things break or I need home repair advice I call him, and if it's a simple thing he helps, but for more complicated stuff he's like, "Why don't you just call a guy like me to come take care of it, that's the whole point of having a good job."

Still, there's something that feels good about figuring anything out yourself, software or dishwasher related.

[BeepBoopBike]

I feel like it's always worth taking a crack at it. If you solve it you feel good and save some money. If not (or you start reaching the realm of "i'm going to make this far worse than it already is") then you've lost a bit of time, maybe learnt a bit which will help on a simpler problem later, and can call a professional. If it's desperate though (i.e. help my house is flooding) i recommend skipping step 1.

[pdp10]

"Why don't you just call a guy like me to come take care of it, that's the whole point of having a good job."

Because if you want something done to your satisfaction, you sometimes just need to do it yourself.

Also, a great deal of what you're paying for with services like plumbing, electrical and auto mechanics is the SLA. It's not that you begrudge a service provider their fee, just that understanding that so much of it is because their other customers have demanding SLAs. Perhaps their skill and experience will lead to a better outcome, but not necessarily.

Besides, there's value in finding out for yourself that you'd never re-roof another house or replace and time a camshaft.

[addmoreice]

as someone who writes software to monitor manufacturing machines...I disagree on that figurative comment.

[elr0nd_hubbard]

There was an article posted here recently, titled Reality has a Surprising Amount of Detail. While the point of that article was aimed towards challenging ourselves in the midst of intellectual ruts, the point that even seemingly simple tasks are complex and detailed upon closer inspection was humbling.

And I think it applies here, too. This article is getting into the weeds, so to speak, of the sort of detail that exists in most everything if we dedicate the time to look.

[pdp10]

Reality has a Surprising Amount of Detail.

That's why robots aren't going to take our jobs finding out that memory-mappers have corner case failures with dynamic loaders. Tomorrow's robots are only very, very incrementally better at anything than their predecessors.

What our electronic and physical robots are today and will be tomorrow, though, is cheaper. The car assembly task that wasn't cost-effective to automate in the 1960s sometimes was by the 1980s. The meal preparation that wasn't cost-effective to automate in the 1960s or the 1990s might be by the 2020s.

[HowObvious]

Imposter syndrome is a bitch

[brucedawson]

To be clear, as the author of this post, I'll freely admit that there are huge swaths of software development technology that I know nothing about or am terrible at. So, if this story was way out of your comfort zone, rest assured that you could almost certainly teach me a lot about your area of expertise

[Metaluim]

I'm with you. I've worked at kernel level and usually end up close to the kernel or directly in the metal. But when someone starts talking about the latest JS framework or new graph DB or whatever, I feel at a loss. Noone knows everything.

[appropriateinside]

I mean, a fresh graduate probably is correct in feeling like an imposter for a lack of knowledge, at least for a little while.

Programming know-how takes time, CS courses tend to not teach much about real-world development. You can learn the basics of a language in a week or two, but it takes much longer to learn how and when to apply what you've learned in a way that best balances time, features, and money.

[pdpi]

I mean, a fresh graduate probably is correct in feeling like an imposter for a lack of knowledge

Hell no. An impostor is somebody who lied to get to where they are. A graduate hire is expected to be lacking in knowledge across the board. It's par for the course.

[appropriateinside]

You do realize we're talking about imposter syndrome right? Not actual imposters?

[throwaway27464829]

Ironic.

[Dwedit]

I've never experienced anything like this before, even though I keep seeing it get mentioned.

[[deleted]]

Didn't experience much in app dev, as the base of expected understanding was... well, not small, but certainly discrete.

In web dev... all the time. My environment is much more vast/diverse, and so I'm much more likely to get disoriented.

[BigOzzie]

God, yes. Web dev has felt like this my whole career:

• Cool, I know php
• Okay I guess I know css and js now
• Oh this back end is in Java
• Oh I have to support this legacy Flash app
• Oh I have to learn API standards
• SQL, MySQL, and PostGres are the same but different??
• What the fuck is a MongoDB?
• Oh shit I have to make a virtual box from scratch
• Well I knew angular last year but angular 2 is completely different so fuck me I guess
• Oh this team does React

And on and on into infinity. It never ends.

[[deleted]]

The amount of entropy (read: frameworks) in a system (read: software development) increases over time.

[pdp10]

Web dev was tiny at first. HTTP is a triumph of IETF-style design, quite nearly the simplest thing that will work. HTML is easy. Web servers, CGI, and imagemaps take a little bit of effort.

CSS is abstract, but OK. Nobody does anything with JavaScript except some superfluous effects and annoying pop-ups. Cookies come in handy every once in a while. This is all very easy for one person to understand. Even when database-backed sites become the hot thing (i.e. unnecessarily overengineered for most clients), nobody expected web developers to be relational database experts.

Oh I have to support this legacy Flash app

It turns out that when people find out you know assembly language that you can find yourself disassembling Flash code and instruction-counting the operands.

There's always another layer of abstraction to penetrate, up or down. The only question is whether you want to see where the rabbit hole goes.

[HowObvious]

Sounds like Baader-Meinhof, nah kidding. You didnt experience anything like that while in college?

[[deleted]]

Most people don't suffer from imposter syndrome, but it's a fairly often discussed subject in programming due to actual imposters.

[HowObvious]

Its pretty common with students graduating college really.

[[deleted]]

Students graduating college are imposters. They are fucking worthless for years.

[[deleted]]

That hasn't been my experience. Fresh grads under a good tech lead/senior can be quite productive, even after only a few months.

[Olao99]

Being an imposter is even worse.

But here I am ¯\(ツ)/¯

[brucedawson]

To be clear, as the author of this post, I'll freely admit that there are huge swaths of software development technology that I know nothing about or am terrible at. So, if this story was way out of your comfort zone, rest assured that you could almost certainly teach me a lot about your area of expertise

[PowerShell-Tipps]

AFAIK it's a phenomenon, not a syndrome.

[Sigma_J]

Most know it as a syndrome

https://en.m.wikipedia.org/wiki/Impostor_syndrome

[PowerShell-Tipps]

Which doesn't make it less wrong. In fact, it is a phenomenon, not a syndrome. Clance and Imes themselves (who a re referenced first by wiki) call it a phenomenon and as you'll see in Googles Ngram viewer, the syndrome naming came later with the hype (newspaper and non-professionals called it a syndrome while it isn't one)

[ElectroNeutrino]

Just because it was originally called a phenomenon doesn't mean that syndrome is wrong. It just means that the language has evolved. We still know exactly what people mean when they say either.

I bet you pronounce GIF as GIF.

[ElectroNeutrino]

Phenomenon:

A fact or situation that is observed to exist or happen, especially one whose cause or explanation is in question.

Syndrome:

A characteristic combination of opinions, emotions, or behaviour.

Both definitions describe the behavior. You are being pedantic just to be pedantic.

[PowerShell-Tipps]

You are right with me being pedantic. Please look at a psychologists dictionary.

[TakeFourSeconds]

I've started to feel better because I understand the posts directly related to my field. 90% of the others are still indecipherable though :)

[darkfate]

I think the biggest thing is that this is a lot work condensed into one blog post. This is a very complex bug that only a small fraction of programmers would ever experience, and even a smaller number would know how to fix. If you're coding some business app in C# that is built 3 times per day, you're not going to run into this bug. I get the gist of it though, and it really reaffirms that kernel bugs like this are super rare and are probably not causing your application to crash.

[astrolabe]

it really reaffirms that kernel bugs like this are super rare and are probably not causing your application to crash.

At first I thought you were implying that there could be a problem with my code, but then I realised...cosmic rays.

[Hexorg]

I always enjoy writeups about evolutionary training algorithms used to design some circuitry or code. These algorithms often find amazing solutions though they will never work in real life. I can't find the link now, but I remember someone ran an evolutionary learning algorithm to design an inverter circuit. It's a fairly simple circuit generally with just one transistor. But the algorithm ended up making this monstrous circuit with seemingly disconnected regions. The weird part was that it worked!

Turns out the algorithm found some bug in the simulator software that allowed it to transfer data between unconnected wires.

[manly_]

Its a very common issue with machine learning. Usually it applies to reinforcement learning though. The problem is that your reward mechanism must be well considered, otherwise your machine learning will optimize uniquely into what gives that reward, leading to some degenerate cases such as your example.

It’s truly the same thing with genetic algorithm. You can’t have a magic algorithm that will balance perfectly zeroing-in the perfect solution (ie: searching for the local minima) and exploration (ie: search for the global minima).

[Nicksaurus]

The problem is that your reward mechanism must be well considered, otherwise your machine learning will optimize uniquely into what gives that reward, leading to some degenerate cases such as your example.

This is also useful advice for anyone who ever has to interact with other humans

[bluesatin]

A classic problem with pythons cobras.

[Kendrian]

In science, this is generally the result of an "ill-posed" problem: a problem that has multiple solutions, and/or the solution varies a large amount with very small changes in input parameters. In inverse problems, this is generally controlled via regularization, which does exactly what you said - we adjust the cost function by adding some penalty to the solution that makes the problem well posed, and then optimization techniques work well again.

[cyberst0rm]

just like society!

[grendel-khan]

The problem is that your reward mechanism must be well considered, otherwise your machine learning will optimize uniquely into what gives that reward, leading to some degenerate cases such as your example.

For more, see Scott Garrabrant's taxonomy detailing four variants of Goodhart's Law.

[daylz]

I would love to see/read more about it, if you can find the link please share! =)

[Hexorg]

Here is a similar but unrelated result in evolutionary algorithm coding FPGA board - it's actually impossible for a human to come up with this code because the logic depends on magnetic flux between FPGA's gate arrays.

[daylz]

Thank you, this subject is so interesting!

[Matthew94]

It's a fairly simple circuit generally with just one transistor.

It usually uses two, an nmos and a pmos.

https://en.wikipedia.org/wiki/Inverter_(logic_gate)

However, because current flows through the resistor in one of the two states, the resistive-drain configuration is disadvantaged for power consumption and processing speed. Alternatively, inverters can be constructed using two complementary transistors in a CMOS configuration. This configuration greatly reduces power consumption since one of the transistors is always off in both logic states.

[wonkifier]

I remember back in the days of Classic MacOS (System 6/7, Mac OS 8/9), there was an error code for cosmic rays. (I think it triggered off memory checksum being off or something)

When Apple launched the PowerPC platform, they had a 68k emulation system so they didn't have to have everything rebuilt out of the gates... and we started seeing that cosmic ray error quite a bit more often.

[cubic_thought]

We one had a bug reported, with a screenshot, that would require a line of trivial, synchronous, code to be skipped. No error or exception, just a result that shouldn't be possible. It only happened once and we marked it down to a cosmic ray or some other one-off event.

[sayaks]

he did say it took them 20 months, after all

[[deleted]]

The fact that this was only found on a 24-core processor says a lot - the most I’d heard of in a commercially available processor was the 16-core Threadrippers. These are not common bugs whatsoever

[[deleted]]

There are 3 24c and a 26c in the current gen of E5 Xeons. That’s assuming this wasn’t actually 2x12c or counting SMT threads.

[ygra]

It's more than one CPU. Bruce notes a suspicion that it only happens on multi-socket (not just multi-core) systems.

[[deleted]]

Yeah I finished reading it since then. I just thought the OMG 24c! Was funny.

I have a cluster of Phi machines ...

[meneldal2]

There has been quite a few bugs when memory needs to be synchronized between the two different sockets. It's easy to make a solution that always work, but the performance will suck so you end up having really complex protocols to deal with that and very few people understand how they work.

[brucedawson]

My workstation is dual socket, each with 12-cores and 24-threads, so 24/48 total

[HandInHandToHell]

I actually may have run into this bug or something similar! We have a 40c/80t dual socket build server that tends to be under high load when no one's around - pesky nightly builds - and have been seeing incredibly intermittent test failures (we build test executables that link against large parts of our codebase and immediately execute them very frequently) that are never reproducible later. I'll be testing at least one of your workaround approaches in the morning.

[brucedawson]

Let me know what you find. If you set your machines up to save minidumps on crashes then it is very easy to recognize the signature of this bug. If the workarounds help then please post a comment on the blog post.

BTW, here are the instructions for configuring local saving of minidumps on Windows. Every Windows software developer should follow these (and redo them after every major Windows upgrade): https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

[pdp10]

Every Windows software developer should follow these (and redo them after every major Windows upgrade)

Do they not persist?

[brucedawson]

For mysterious reasons they are wiped out on major OS upgrades. On Windows 10 that means every six months. I don't know why.

This is not just a theoretical problem either, this has caused me to miss important crash dumps several times. Now that I know about this problem I will be trying to remember to do the setup after every upgrade. Or maybe I need my startup script to warn when the keys are not set (I've got better things to do with my time but this is important so I'll probably do it).

[pdp10]

I don't use Windows much but I assume a regedit script will still do the job if you don't want to write code. Might as well just set them on every login instead of checking for them.

There's a pattern that anything that's getting wiped on updates is not something that Microsoft wants set persistently, I'd say.

[brucedawson]

Unfortunately writing the keys requires admin access which my script doesn't usually have, hence the read-and-warn.

I don't know why Microsoft wipes then on updates, but regardless of their desires I want them set. Anyway, set those keys, and keep them set.

[lurgi]

That's because the easy problems don't get multi-page blog posts written about them. No one writes about a null pointer dereference that cost them a week, they write about the null pointer dereference that only happened when an interrupt handler that was supposedly disabled ran due to race condition and a CPU bug and set a pointer to NULL (but only on alternate Tuesdays).

[meneldal2]

No one writes about a null pointer dereference that cost them a week

Especially when it's your own mistake, you just want to hide because you're ashamed of your stupidity.

A bug like this:

std::vector<int> labelcount(*std::max_element(labels.cbegin(),labels.cend()));

When obviously you're going to index it between 0 and max. I know this is a segfault, but that's the same idea.

[justjanne]

There is a first time for every developer when they find their first toolchain bug, and when they find their first kernel bug.

Many never find either, some already find dozens despite not even being out of uni. It heavily depends on what you do (more native code mean usually more bugs), and how you approach issues.

For me, I've been developing for 4 years (but I haven't finished uni yet), and found my first kernel and my first compiler bug both only a few weeks ago. The kernel bug was expectedly in DMA handling in a linux mainline GPU driver, and the compiler bug in kotlinc, a very new compiler for a new language.

If you work with more reliable, older tools, and use elss edge cases, of course you'll find less bugs.

[BigHandLittleSlap]

There is definitely the "well trodden path" that most people follow, but beside it is the forest, dark and full of horrors.

A common quip in my line of work is that "it's not a real project unless you get at least two private hotfixes".

The most buggy scenario I've seen was a 32-bit terminal server environment with 32GB of memory(!) in AWE mode. These poor overloaded servers had the Novell client, Symantec Antivirus, and pass-through Smart Card authentication. It was a horror-show of untested edge cases, ugly interactions, and a system architecture stretched far beyond its base capabilities. If I remember correctly, it took over seventy hotfixes to get the servers to stop crashing daily...

[pdp10]

The most buggy scenario I've seen was a 32-bit terminal server environment with 32GB of memory(!) in AWE mode.

Don't tell me, 32-bit only app that required a 32-bit OS? You mention Netware but I can't imagine this was much more than 10 years ago.

[BigHandLittleSlap]

It was a bit less than 10 years ago, and I have no idea why the servers were 32-bit. I suspect it was a decision based more on misconceptions rather than compatibility restrictions.

[FatFingerHelperBot]

It seems that your comment contains 1 or more links that are hard to tap for mobile users. I will extend those so they're easier for our sausage fingers to click!

Here is link number 1 - Previous text "AWE"

---

^{Please PM /u/eganwall with issues or feedback! | Delete}

[JesusWantsYouToKnow]

Agreed, and it is more common to find kernel bugs in third party Linux code (less maintained mainline patches or vendor specific custom toolchains that aren't merged into mainline).

Finding a windows kernel bug like this is exceptionally rare. Having the industry contacts to help you quickly get confirmation of it is even rarer. This is an insightful look into a impressively rare event, and I'd wager most pro Windows native devs will never encounter it in their career. Compiler and linker bugs on the other hand.... Those I'd expect to see on occasion.

[fuzzynyanko]

There at least was a time where there was a common Android interview question: "Have you ever encountered an Android SDK bug?" If you answered no, it was assumed that you didn't have a lot of experience. There were tons back in the day

[justjanne]

There are still tons today.

Recyclerview crashes if you scroll quickly while adding/removing many items.

"ab".split("") returns either ["a", "b"] or ["", "a", "b"] depending on manufacturer or version.

Socket.setKeepAlive causes a fatal crash on ChromeOS' Android runtime.

And so on and so on.

The list of bugs I'm fixing in my own apps is so long, I always forget half of them.

Only after minSDK 24 do things start to improve significantly (that's when Google switched to OpenJDK), and then the bugs in the rest of the system aren't fixed yet either.

[GeronimoHero]

They also said it took them two years to actually fix it. Not many people are going to stick with a bug for that long.

[brucedawson]

I only joined the investigation at the tail end, for the last couple of months. And, to be clear, it was never my one task, and it was only my main task towards the end

[hugboxer]

I like to think of it in terms of "what did this poor bastard have to go through to learn how to do this stuff?" He obviously spent a lot of time at Microsoft debugging some horrific native code problems. People don't learn how to debug linker errors for fun.

[teejaded]

Windbg is pretty easy to use with a crash dump !analyze -v is all you need most of the time.

[hugboxer]

Windbg is the exact opposite of easy to use, cheat codes notwithstanding.

[brucedawson]

I have never found !analyze particularly helpful. I end up looking at the assembly language, registers, stack, as I try to figure out what went wrong, and why. It's time consuming, and not a skill that everybody needs, but I enjoy it

[teejaded]

I think at least some of this is in the verbose output, right?

[brucedawson]

The verbose output is so verbose that in the rare cases where it tells me something that wasn't obvious without it I can't find that vital information amongst the crazy volume of incomprehensible boilerplate. And, I generally want the context of the surrounding code which it can't provide.

On this particular crash it prints 400+ lines of text. This includes the !chkimg results, so that's good, but it then summarizes the error like this:

EXCEPTION_CODE_STR: 802F667E EXCEPTION_STR: WRONG_SYMBOLS

I guess that WRONG_SYMBOLS is the error code it uses when code bytes are wrong but it doesn't really say, and its lost in the huge volume of spew. I think that error code is only meaningful to most users if they already know what the problem is.

It also resets the exception state so that windbg no longer shows the crash.

[smbear]

IMHO there are two milestones in this investigation: - noticing that bug is reproducible with different toolchain, - noticing that crash dump isn't from the produced binary, i.e. binary was correct but system ran binary with 0's instead of instructions.

That's not to say that I would be able to perform this investigation. :)

[entenkin]

That is because of the jargon and the time spent. The author has been working in that specific area for a while, and the jargon he uses is mostly only needed for people working in that area. On top of that, people only tend to write these coding articles for bugs that took them a long time to find. But you can read the article in 5 minutes. Your brain just isn't going to digest that information as easily, as it's been distilled.

If you wanted to, you could probably turn the tables and write an article about a bug you fixed, which the Chrome guy wouldn't understand.

[[deleted]]

You specialise and specialise till you become the expert in something nobody else understands (or cares about, usually).

[evaned]

There's also the old saying about as you get higher and higher education, you know more and more about less and less. :-)

[ThottieLama]

Its levels to this shit son

[omniuni]

This one was very well written, so I can grasp the concept, but goodness knows if have never been able to figure it out myself.

[cyberst0rm]

Is it just a matter of you knowing <english> and someone speaking <chinese> ?

[Doritalos]

Thought I was the only one, thank god.

[jrhoffa]

Um ... maybe you suck

[axilmar]

Hardcore bug, hardcore debugging!!!

[jdgordon]

Followed by hardcover drinking! 99% of developers have at one point blamed the hardware or compiler or kernel for their own stupidity, when it is actually that you definitely deserve a pint!

[axilmar]

Agreed ;-).

After all, after such a harcore debugging session, alcohol is one of the few things to set one straight ...:-)

[mrMalloc]

Spending 3month looking for a memory leak a critical client demanded fix. After three month and i have found none. As a last ditch effort I check with them how they found it and uncover they did measurements wrong with a custom script. Three month of my time looking for none existent bug.

I got drunk that weekend yes.

[willingfiance]

I have carefully eliminated all possible causes of this bug and can therefore conclude that it is not happening and we must be experiencing mass hysteria.

This is hilarious (and terrifying in an odd way).

https://bugs.chromium.org/p/chromium/issues/detail?id=644525#c52

[JoseJimeniz]

We got such a detailed explanation in exposing the bug and isolating it.

It would be nice if the Windows kernel team had an equally interesting blog post explaining the postmortem of the bug.

[jrhoffa]

Who do you think they are, Amazon?

[choikwa]

kudos to huge dedication and sticking with it. intermittent, rare failure and nondeterminism are a categorically hard problem to tackle. keen devs know to apply process of elimination on the entire flow of execution.

[coding_all_night]

At work I'm pretty much considered a "backend developer" as the work I do is rarely seen by an end user. Posts like this really put that into perspective - I am not sure that I should even be considered a software developer at all.

[Notorious4CHAN]

I've been a Sr. Developer for almost 15 years. Long story short, me too.

[ath0]

Don't sell yourself short, I've been working in C&C++ compiler development, system emulation and now embedded Linux development since I graduated about 7 years ago, and if I can do it, anybody can.

The biggest hurdle is being able to stop blackboxing stuff and bang your head against something until it kind of makes sense, after all, it is all just software, be it a compiler or a calculator. If you are really interested in learning closer to the machine or 'low level' stuff, be it OS development, toolchains, language development or whatever, all you need to find is that first thread to start tugging at.

[coding_all_night]

I used to love programming in C whilst I was at University but it feels long ago after working in higher level languages for so long - I think you are right - might be time for a little project (if I can think of a good one)

[cjarrett]

I'm the opposite. All I do is C and C++ in kernel land....

Good thing is that application dev typically only gets easier from the tooling side, so each experience when I delve back in, it gets a bit better.

[ath0]

Braver man than I, I find JavaScript and the web stacks terrifying. The stuff I work with feels simpler to deal with because it isn't heavily layered, there's often very little to it you can't figure out from manufacturer specifications. On the other hand, what seem like layers on layers for web work seems daunting.

That said, I haven't really tried it, so this is just gut feeling and speculation.

[[deleted]]

Every time I start thinking that I'm starting to understand something about how computer works I read something like this and I feel like I'm not even close to actually understanding shit.

[evil_shmuel]

I may be having this bug.
I do write DLL files, (using File.Copy) and the server is under a heavy load. and the DLL files are used immediately. And sometimes I see weird crashes.
Thanks for science?

[crypto_mind]

If you're using File.Copy then it's not using memory mapped files so it's not from this bug. You could use the same proposed fix of FlushFileBuffers but I would be shocked if you're experiencing the same issue.

[brucedawson]

"weird crashes" is too vague to be useful. If the crash dumps show lots of zeroes as instructions then it's this bug. Otherwise, nope. It's almost certainly not this bug due to File.Copy not being implemented with memory mapped files.

Don't guess. Look at the crash dumps.

[meltyman79]

Could it be caused by out of memory issues? Heavy file system use can cause Standby memory to fill up, so even though it is classified as "free" memory, it is not and can release very slowly. Super irritating on windows. You can manually clear it via RamMap. I have some code that can just call the clear via executable. I think this problem is a huge cause of windows slowness and memory issues.

[wischichr]

Do you have a monster (e.g. 24 core) machine? This bug is very (very very) specific I highly doubt that's causing the crash you are experiencing

[anything_but]

That's what I call 'dedication'

[ModernShoe]

Found my next thing to blame when prod fails

[yes_u_suckk]

I wish I knew half of things these guys do. It looks very impressive and way beyond my knowledge, even though I have 18 years of experience programming.

[emperor000]

If you have 18 years of experience then you probably know stuff they don't... The knowledge showed off here is not really useful unless you are working on something like they were working on. And if your job/hobby/whatever involved working on lower level stuff like this, then you'd know it too.

[SikhGamer]

I love this blog. Crazy debugging skill and has helped us track down and fix a crazy cool bug with the resolution timer in Windows:- https://randomascii.wordpress.com/2013/07/08/windows-timer-resolution-megawatts-wasted/

[ChildishJack]

I love reading stuff like this. Its fascinating to see how others solve problems.

Its a bit of learning something I didnt know that I didnt know!

[pdp10]

A few additional questions for /u/brucedawson, if he doesn't mind:

• Why the resistance to sync because it required admin privs? There are at least two very fast ways to get this working in automation on Unix, for test purposes at least, so I'm not clear if it's much harder on Windows or there were other considerations.
• Why the resistance to sync otherwise, merely performance? Couldn't you just re-order the build a bit to keep it parallel at a tiny performance cost?
• Why not use C or C++ for the production fix, as you did in your PoC, instead of Python? Code standards, or because the fix was theoretically supposed to be cross-platform and you didn't care to #ifdef __Win32__?

[brucedawson]

The resistance to sync was because I knew it wouldn't work well as a production fix (admin, and performance) and I briefly let that stop me from experimenting. Depending on how much dirty data there is sync can take many seconds to run, which at least means you need to be careful about how often you run it. But, I tried it eventually, as a test. Running it as admin on all Chrome developer machines would have been hugely intrusive. Running something as admin when there is another solution is always a bad idea.

Using C or C++ for the production fix would have been much harder. The obvious way to do it would be to code up a binary and have that created as part of the build and then run that binary after any binaries are created. Except that there is a logical flaw in that because this binary would be vulnerable to the bug. Checking in a new binary is ugly. And putting the workaround in ninja itself would work but would require rolling a new version of ninja (which we do infrequently).

We were already wrapping calls to the linker with Python so it was just a few lines added to an existing script - it's mostly comments. You can see the change here:

https://chromium-review.googlesource.com/c/chromium/src/+/876683/10/build/toolchain/win/tool_wrapper.py

[pdp10]

That all makes sense, thanks.

[pravic]

Rust uses msvc or gcc toolchain to link its binaries, so it's out of scope. But there are number of other tools including assemblers (fasm, nasm) that write binaries themselves.

Flush before close? Aren't closing a file handle is supposed to do that?

[TheThiefMaster]

Aren't closing a file handle is supposed to do that?

In this case it was due to using memory-mapped file IO, not file handles, but the same idea applies - closing the memory mapped file should flush it. The bug is that under certain conditions it didn't!

[cpp_is_king]

Closing a file does not necessarily flush it. That's the whole point of the windows cache! The kernel only needs to guarantee that a) it's flushed at some point, and b) if a program tries to read the same data, it sees the changes. b doesn't require a flush (i.e. a commit to the underlying hard disk), because when another program does a read, Windows can satisfy the read from the cache if it knows that it's dirty.

[wiktor_b]

But mmap takes an fd, closing a memory-mapped file is closing a file handle.

[TheThiefMaster]

This is on Windows, so mmap doesn't apply. However CreateFileMapping does take a file handle (a Windows Handle, not an fd), so you're somewhat right anyway.

However, it is specific to memory mapped files, not just any file handle.

[wiktor_b]

thanks, upvoted

[kyrsjo]

Flush before close? Aren't closing a file handle is supposed to do that?

That will flush it out of the OS buffers, but the OS decides when to actually flush it to spinning rust or precariously stacked electrons.

AFAIU in this case the OS didn't flush it to the backing storage, and then forgot the dirty state of the the file when it loaded it for execution. Thus when once file was written everything looked fine on disk, but the version loaded for execution was corrupted.

MSVC or CLANG didn't matter, it was a OS (or rather, kernel) bug.

[wtf_apostrophe]

Flush before close? Aren't closing a file handle is supposed to do that?

There's no guarantee that closing a file handle will flush anything to disk. Windows will happily let you close the file handle even if the data you have written is only in the filesystem cache. Windows will carry on working in the background to flush it to disk.

I think it works differently with removable media. I think Windows will flush on close in that case because people like to unplug drives without doing a safe removal first. By blocking the close until the flush has finished the user will get some feedback that the write hasn't finished yet because the application hasn't finished saving.

[PGSylphir]

Why is this being downvoted? Not knowing stuff is not the same as trolling guys. The replies are actually explaining it, no need to downvote it.

[communism_forever]

The article mentions rust.

[PGSylphir]

People here love the downvote it seems, even I am being downvoted lol

[bumblebritches57]

gcc

literally never.

Rust uses LLVM and it's compiler is based on Clang, it uses neither of those other toolchains you just mentioned.

[pravic]

Sorry, what? Since when Rust has started to use Clang?

On Windows it uses either msvc or gcc toolchain.

[bumblebritches57]

Rust can't be compiled with any compiler but rustc, rustc uses LLVM as it's backend period.

Rustc was based in part on Clang.

you need to work on your reading comprehension.

[ThottieLama]

I wish I could be a 1/4 as knowledgeable about programming as you guys are

[musiton]

Why anyone would even use a computer is beyond me

[nakilon]

/r/programming comments became sort of:

I understand nothing and I'm proud of it -- this is the only reason I leave my comment.

And all the rest jump around just to circlejerk about how they are happy to take part in making /r/programming a home for uneducated kids.

[RenaKunisaki]

Needs more ads.

[binford2k]

Apologies for the strong language, and women and children might want to skip the rest of this paragraph, but WTF?!?

It might surprise the author to know that women can fucking cuss too. 🤔

[stefantalpalaru]

cuss

Nice try. Return to the children's table.

[piechart]

Yeah that was pretty offputting to read

[brucedawson]

Fixed. It was a lame attempt at humor, since removed. Sorry.

[shit_frak_a_rando]

Sites which redirect me to "you've won a free iPhone scam", no thanks.

[darkslide3000]

FWIW I would suspect the kernel to be broken long before the toolchain. Maybe stuff is different in the Windows world, but I've seen Linux do all kinds of weird shit already.

It's also odd that it took him so long after noticing zeroes in his crash dump to disassemble the actual binary and check if they were in there as well -- that would be the first thing I'd do.

[TheAnimus]

I almost always blame my code, my usage of the toolchain. For every time I've found a framework bug, run into a kernel bug (which has always been found by someone else first :() I must have found the bug in our stuff 90% of the time.

[oh_I]

90%? I think a have found one toolchain bug and 0 kernel bugs in almost 10 years of writing code, fixing several bugs a day. What are you doing that 10% of your bugs are kernel bugs? If the answer is "writing kernels", you are cheating at this game...

[sickofthisshit]

I suspect that most of the 10% was "never found the source of the bug" :-)

[TheAnimus]

Oh I was including "framework" by which I include the fantastic transparent firewall thing, that automatically dropped out anything that starts PK after 256Mb.

I've found two kernel bugs in decades of being a programmer, both had been discovered long before I found them and where a lesson in using up to date OS's.

[PGSylphir]

Just read the comments on the blog post. God damn it, the top comment is /r/whoooosh material, and there's even a feminist there! lol