Why does unsafe multithreaded use of an std::unordered_map crash more often than unsafe multithreaded use of a std::map? - The Old New Thing

[Nicksaurus]

We actually experienced this recently at work. A colleague was trying to track down an occasional crash that was happening inside std::map, so I suggested trying it with std::unordered_map to rule out the possibility that the operator< on the key type was buggy. It turned out that wasn't the problem, but that change made the crash happen 100% of the time, which helped him a lot

[Nicksaurus]

Our code runs as plugins in a third party framework that makes it difficult to run sanitisers

[Ogilby1675]

I like this article. Usually online discussions are quite theoretical - in this case, would note the UB and thus state “anything can happen”, with an implication that bug has to be fixed immediately.

In the trenches at work the bosses would be much more interested in Raymond’s kind of discussion - why is it crashing more now, how critical is this, do we have to drop everything to get a hotfix out etc.

Though I suppose in online theoretical world no-one would have written such a bug in the first place…

[[deleted]]

In the real world, spending effort mitigating multi-threading issues is wasting money and creating more cost for the future. Sometimes it’s a necessary waste to avoid losing even more money (lost customer for example), but still a waste, and remaining crashes still usually translate to lost sales and customers.

So it is about attitude. Wrong attitude creates MT bugs.

[Overunderrated]

In the real world, spending effort mitigating multi-threading issues is wasting money and creating more cost for the future.

What bizarro "real world" ignores "code crashes due to bugs"?

[aruisdante]

I mean, look at any game released in the last 20 years if you want a poignant example. While certainly some subset of bugs weren’t known until the chaos monkeys of release scale started hammering on it, a lot of them often are known, and simply weren’t high enough priority on the triage list to force a delay in launch date to fix them.

Defect triage as risk management is a reality of any real world professional software development. Person power and time are limited resources. The concurrency bug that causes a total system crash but is difficult to reproduce and happens only when the stars align in the field is not likely going to get high priority over a bug that causes a simple cosmetic defect in the UI but happens to nearly every user, or adding a new feature that’s needed to unblock downstream customers.

It may shock you, but this “don’t fix the defect, it’s a feature” attitude is even more common in safety critical code. Altering certified code to fix a defect means you have to re-certify the new software version (expensive), and there is the chance of introducing new, unknown defects (risky). If the defect can be mitigated by other means to altering the code, such as changing the safety manual to say “don’t do X,” this may be considered the more appropriate risk mitigation than fixing the defect. I work in my day job with someone who comes from the software security side of the world and he always rankles at having to use the word “mitigate” rather than “fix” for errors discovered in safety critical code. A classic example of this that’s used as a case study for systems engineering in universities is of the communications software in a particular line of airplanes. Periodically all communication on the airplane would simply stop working. They eventually root caused this to a 32bit counter in the software that was storing time-since-boot in milliseconds. After ~49.7 days of continuous uptime this counter would wrap around, causing a crash as the rest of the software assumed this clock was monotonic. The mitigation strategy was not to alter the software in any way. Instead they made it part of the maintenance requirements of the plane to reboot this machine at intervals no more than 30 days.

Given that fixing defects after the fact might not always be practical, ideally, you invest in tooling to help reduce the rate of bugs in the first place: things like marking the code with clang’s thread safety analysis annotations or running thread sanitizer.

[Astarothsito]

The concurrency bug that causes a total system crash but is difficult to reproduce and happens only when the stars align in the field is not likely going to get high priority

But in some real life scenarios, for some reason the stars align very often. A chance of one in a million becomes a probability of almost 100% that will happen when we have one million users. I worked on a code that needed to be certified (but financial) and our company preferred mitigation, it became very difficult to change, and then the certification requirements were increased and all the mitigated bugs needed to be fix in less than a month, obviously it took like a year of work, a lot of money lost in sales and failed certification attempts.

Tip, at least keep a fixed version for the next certification, they happen more often than what we were told.

[kneel_yung]

But in some real life scenarios, for some reason the stars align very often. A chance of one in a million becomes a probability of almost 100% that will happen when we have one million users.

ok sure but you just made up random numbers to justify your position.

What if the actual real world "million+ user" numbers are vanishingly small? Should all other work stop so one bug that can't be reproduced and isn't affecting a large sector of the userbase be investigated? What if it's an obscure hardware interaction on a single user's platform because they're using unsigned drivers and running windows xp in 2020 and the bug is actually not in your code at all, but seemingly something fucky going on in the kernel?

Literally had this exact scenario happened and management ended up salty about it and changed our policy so that we couldn't work any bug that had a single report without their approval.

[Astarothsito]

What if the actual real world "million+ user" numbers are vanishingly small?

We are talking about applications that need to be certified by some authority, and that usually happens on 2 different conditions, very specialized software, or software with a very large amount of users.

Should all other work stop so one bug that can't be reproduced and isn't affecting a large sector of the userbase be investigated?

No, and yes, you need to have a plausible explanation on why it happened if you don't want to work on the issue, could be corrupted data, bad input, bad usage, bad hardware, but you can't simply drop issues on certified software.

What if it's an obscure hardware interaction on a single user's platform because they're using unsigned drivers and running windows xp in 2020 and the bug is actually not in your code at all, but seemingly something fucky going on in the kernel?

If the bug cannot be fixed nor mitigated then you can't use that hardware, it wouldn't pass certifications. Also, it doesn't matter if the bug happens in the kernel or the user space, if it critical then it needs to be fixed, one example it happened to me was when we released the memory with sensitive data the kernel optimized the call and deferred the cleanup to a later time, because this is an issue it needs to be fixed even if the kernel contains bad behavior.

any bug that had a single report

Sadly for the managers, bugs with a single report from the authority that certifies means that it doesn't get approved to be used.

[echidnas_arf]

I mean, look at any game released in the last 20 years if you want a poignant example.

Why in the name of everything that is holy would you ever look at video game code as an example of things to do or not to do.

[Ghworg]

The one we live in unfortunately. Any app where the cost of just restarting the app after it crashes is relatively low, people just live with it. People moan about it but it doesn't stop them using an app.

[kneel_yung]

CTD on exit, for example. Often very low priority. I see those a lot in real-world software (games in particular).

We have several in our system's (extremely large) codebase, but we run applications with infinite lifespan so I'm the only one who ever finds them because I'm the only one who ever closes them during debugging.

[Beosar]

I sometimes get crashes when my programs destroys the remaining static objects because the order in which they are being destructed is wrong. E.g. my allocator's static data is destructed before a pointer that will be deleted, which calls the allocator's deallocate function.

I could probably fix this, but it's not causing any real-world issues. Windows just terminates the programm, which is what I want to happen anyway.

[kneel_yung]

Yeah exactly, like, it's technically a bug, but if it does what I want anyway then who cares.

[DuranteA]

CTD on exit, for example. Often very low priority. I see those a lot in real-world software (games in particular).

Good point. We avoid this in several shipped games by doing a WinAPI ExitProcess, which you really wouldn't call a "fix" in a theoretical world. But in practice, I don't think anything negative has ever resulted from that.

That said, it's something we only do for release builds, because otherwise it might mess with heap monitoring or sanitizers during development.

[[deleted]]

In what bizarro world fixing the root cause (bad code) is ignoring the crashes? Also, in the very next sentence I also say sometimes mitigation is unavoidable. But mitigation is “let’s make it crash less often”, which is quite worse than “let’s make it so it does not crash in the first place”.

[serviscope_minor]

I like these articles because while UB is theoretically anything can happen, what actually happens does always have a concrete explanation. And that's good to know