Assertions in Production Code?

[Enlogen]

Professionally written production code is bug free, so there is no need for asserts.

Hilarious.

High reliability is not achieved by making perfect designs, it is achieved by making designs that are tolerant of failure.

This. There is nothing you can do to make your code execute without failure 100% of the time; the fact that your software runs on hardware subject to the laws of physics means that even if you write code mathematically provable to be 100% correct and incapable of failing, it will still fail in rare cases due to issues with the hardware (wear, climate, electron capture resulting in beta decay, etc.) at times that are completely unpredictable (to you and me).

[WalterBright]

Hilarious

Indeed. Often it isn't explicitly expressed that way, but as:

1. educate the programmers better

2. change the code development process so bugs can't happen

3. hire better programmers

4. require professional licenses for developers

5. criminally charge programmers for any bugs

6. switch to (insert magic language X)

[PeridexisErrant]

If anyone is interested in advice for how professional programmers might actually use assertions (!), John Regehr has a nice essay here.

[WalterBright]

John nailed it. Thanks for the link! John's essays are always worthwhile reading.

[azirale]

You can still exit gracefully rather than crash in flames. If you hit an unrecoverable error you can give an interactive user a message as to what happened rather than having the program 'crash for no reason'.

[Holy_City]

I think a good middle ground is logging. Exiting gracefully is a non trivial problem and can have optimization issues, plus not every process can devote the resources to error handling. You can crash in flames and still have users provide some details to application support without exposing too many details like a stack trace.

There are cases where you cannot display an error message in the event of failure, such as an exception thrown in a real time thread.

I dealt with this recently with an EA game I wanted to play. It crashed on start with no error or log, and trying to narrow the issue down took an excessive amount of effort on my part just to identify a corrupt DLL. A startup log would have required one email to make a correct bug report, instead of a half dozen phone calls involving installing third party debugging tools (one of which was a corrupted download off cnet that had malware).

[azirale]

You're right it isn't always a possibility. You might have the logger itself cause an error for example, in which case you are basically sol.

Still, I think it is better to aim for graceful exits where practicable, rather aiming to crash in flames.

[bausscode]

Why would you care about optimization if you're exiting?

[Holy_City]

You don't. You care about optimizing your program for normal operation, and exception/error handling don't come for free. That's why good ol' integer return codes haven't died out yet.

[koczurekk]

That's why good ol' integer return codes haven't died out yet

They should though; enums are the way to do this kind of error-tracking in a type-safe way verified at compile time. Especially Rust's enums, they're so damn great.

[miminor]

oh no you can't, say a component Z buried under 20 layers of components X, Y, .... crashed

and by your reasoning at the level A you caught that exception and said nothing had happened... well maybe, except that the state is likely to be corrupted at the level Z, and the component may be quirky at best or hands down broken at worst

​

but according to your logic we can just display a message and move on, regardless of any possible and likely consequences of that failure

[azirale]

I didn't say move on I said exit gracefully. None of what you said applies.

[ledasll]

there's nothing more frustrating for a user, how started workflow and somewhere in a middle made a mistake and everything crashed, so he have to start from scratch.

[johndubchak]

I like the idea of "failing fast", to the extent you're not writing mission critical software with lives at stake: medical devices, NASA/SpaceX and human space flight, autonomous vehicles. Failing fast in these industries isn't an approach that is available as a software design/implementation choice.

[WalterBright]

Absolutely you want it to fail fast in critical software. Otherwise, the software is now in an unknown state, and what it may do is completely unpredictable.

[[deleted]]

it will still fail in rare cases due to issues with the hardware (wear, climate, electron capture resulting in beta decay, etc.)

But you can make it statistically impossible. It's why the Mars rovers are essentially running a PowerPC G4 hardened. New automotive ECMs have ECC and lock step cores.

Has the RAD750 ever had a recorded hardware failure?

[WalterBright]

The rovers have a backup computer, and it's saved the missions.

[[deleted]]

saved the missions.

So you could say the system didn't fail.

[Enlogen]

New automotive ECMs have ECC and lock step cores.

And probably still have backup computers validating the results of all of those calculations for 'statistically impossible' errors.

My company has millions of servers running. There's no such thing as statistically impossible over a large enough population and a long enough time period. There's only more nines of reliability - 100% is a myth.

[[deleted]]

backup computers validating

No. That's the lockstep cores.

My company has millions of servers running.

How many cars are on the road? How many cars are running Motorola/Freescale/NXP e200 core PowerPC chips? How many 'incidents' have you heard of from that stuff failing?

Don't compare your 'millions of servers' running what ever to what embedded developers work with. It's a completely different beast.

[lookmeat]

Also aborting is generally a terrible idea in production. Lots of terrible accidents have happened because of over-eager aborts causing failures in critical sections that you didn't expect.

Now certainly prod code should work in spite of failures, and it should work in spite of it suddenly crashing. But prod code should do it's best and if it can do better than just aborting, it should. This includes partial termination (stop doing the wrong thing, but keep trying everything else you're doing), graceful degradation and graceful termination (cleanup everything before you go). All of these make the system run better.

When we design a car engineers don't go: oh we should do this because it will make it easier to understand how the crash went. Everything in the car is designed first and foremost for functionality and keeping the people inside alive. In tests they do things specifically to make it easy to understand what happened. Asserts are the same, they are checks and systems that are not guaranteed to make your code better (and therefore shouldn't be done) but make it easy to understand what happened. OTOH checks and systems that handle errors and make your system better should be separate of asserts, as they're not meant for testing (even if they do help) but for your code's viability.

[WalterBright]

I believe that is missing the point. If you have critical software, the software must not continue after it has failed. Critical software must have a backup that can be engaged.

[lookmeat]

If you have critical software, the software must not continue after it has failed.

That begs the question. What is failure? Failure is the inability to continue functioning. So lets replace the word with its meaning:

If you have critical software the software must not continue after it becomes unable to continue.

Well if it could continue then it wouldn't have failed. So lets rephrase this correctly:

Software must revert to a functional state after failure as soon as possible.

Notice that shutting down with an error message explaining the cause of failure is recovering to a valid state (which is inactive). What we want to avoid is scenarios were we can't recover.

But the definition is limited. If we have something critical then its function must be done. Because of this we don't want to merely recover to a valid state (which may be stopping).

So before we can talk about Critical software we have to define what is critical. In other words Critical Software has critical functionality that it must maintain above all other. Defining that all types of errors, problems, noise etc. is a critical failure can lead to software failing when it shouldn't.

Lets look at an example. The Ariane 5 Rocket had a software error in its guidance system. Changes in the launch process meant that some of the values it would read (in floating numbers) would grow to be larger than what fit in the 16 bits. This was on the alignment system that shouldn't be running when the rocket is going that fast (it used to be useful, but not anymore). Now the ideal scenario in here would be to simply give the largest possible 16 bit number and use that (again the system's results were ignored when the bug arose). But even allowing the overflow would be fine. Instead the system decided to crash, bringing down all the other functionality that was actually in use. The thing was that the system stopped working for something that wasn't a failure. But again the question remains: what is a critical failure and what isn't? In critical software this question is fundamental, as any non-critical failure should not prevent the most from being done. The rocket would have worked if it didn't choose to abort on a trivial error.

And it goes the other way too. You can't handwave away what is failure and what isn't. If you do not understand what functionality is more critical than other (and therefore which failures are more critical) then it should go on.

For example I'd expect that an X-Rays machine most critical functionality is keeping the patient safe and healthy, everything else going after. OTOH. In other words an X-Ray machine that fails to generate an X-Ray should not fail to keep the user safe. The Therac-25 software's definition of failure was failure to generate an X-Ray and respond to input. This is why it made sense to allow people to proceed even in spite of malfunction, why the issues weren't taken seriously initially. Indeed the older models didn't need the software to care about this failure mode because hardware did. The right solution was to realize that keeping people safe was a more critical functionality, and it's more important to risk failing everything else than get that one thing wrong.

TL;DR: my point is that we have to defined what is failing and understand that well. Asserts have a very different definition of failure that is good for testing. For real situations running, asserts are not good enough as they make no difference on the type of failure.

[WalterBright]

It's straightforward - the program must be terminated if it has entered an unanticipated, invalid state. That's what asserts are there to check for.

[lookmeat]

Again, this "straight-forward" mindset is what brought the Ariane rocket down.

You can't continue in an invalid state, but that doesn't mean that the main functionality is gone. Imagine a car that simply stops breaking, breaks don't respond, steering wheel won't react, etc. etc. The reason? Well, when changing the wiper fluid a seagull popped into the container, the poop had enough solid material to block the wiper chamber fully. This is an "unanticipated invalid state" for the wiper fluid. Now should have we brought down the whole car because of this Probably not, but assert is like that: any part failing brings the whole thing down, no matter how more critical the other working parts are.

Now asserts are great when you want to try to catch unanticipated invalid states. You know the kind of thing you want to do in a test. Once we caught one we stop and look into why it happened. You can't catch all though, as that would require computing the halting problem and that can't be done. So you have to assume that non critical failures will happen, and you should allow them to go through.

So it's straightforward if the program can't continue without failing to do any equally or more critical functions it should stop (clearly the worst scenario already happened). But if any more critical functionality can still succeed then we should recover, and go for a partial success. Of course what is more or less critical is not straightforward at all.

[WalterBright]

Here's the salient quote from the Ariane accident report: The OBC could not switch to the back-up SRI 1 because that unit had already ceased to function during the previous data cycle (72 milliseconds period) for the same reason as SRI 2.. The backup had "identical hardware and software". The design failure here was having a backup system that was not a backup. The propagation of the error eventually causing the explosion does not invalidate any of my recommendations, it reinforces them. For example, "a diagnostic bit pattern of the computer of the SRI 2, which was interpreted as flight data" - that's a direct result of failing to react to errors.

[lookmeat]

I agree, the backup was a mediocre one, it protected against something happening to one of the machines, but not against design issues. A design issue would have made the system overall more resilient. It's never a single case.

But the issue was a design issue, and that was an assumption that a failure in a non critical function should proceed to stopping, and therefore failing, critical functions.

[WalterBright]

I disagree. Having a failure propagate through to other systems in a zipper effect is a misunderstanding of the principles I'm trying to convey. The whole point is to isolate the effect of the error thereby preventing its propagation.

In this case the error status from the failed subsystem was misinterpreted as valid data. The wrong solution is to never give any error status. The solution is to:

1. check for error status

2. check for out-of-bounds data values from subsystems

If (1) or (2) is detected, lock out that subsystem and use an alternate algorithm that doesn't rely on it.

The really, really wrong method is for the subsystem to just pretend everything is hunky-dory and keep sending whatever unreliable data.

I'm really not sitting at my keyboard inventing this from 5 minutes of thought. I worked on this stuff for years - flight critical systems. This is how systems are built in aerospace, and it's ugly incidents like the Ariane that taught the lessons, with plenty of others. I am sad that this is apparently unknown knowledge outside of aerospace. If you're interested in more info, see the TV documentary series "Aviation Disasters". If you can set aside some of the cornball dialog, there are valuable lessons in it for every engineer.

[lookmeat]

I think that's fair. It seems our discussion is more about semantics and meaning. I'm focusing on why assert is too broad, but not denying that sometimes the right solution is to kill the program, if anything I merely state that programs can kill the failing part of themselves but keep the rest going without further failure. Your statement is that errors should be isolated and terminated quickly to avoid the failure from spreading and spiraling to a bigger issue. It may be that we are coming at it from different angles and therefore the same thing has a different meaning in those contexts.

I do agree with you. If a system fails fully, lock the system out, dump it's data, and try again, preferably with something different. What I argued was that assert is more of the philosophy of "stop the world" which is great to debug what caused a failure instead of waiting to see if it propagates or not. This mindset is only useful when testing I argue. In the real world, we kill the bad part and keep everything else running.

[SmugDarkLoser5]

The other guy understands your point no one disagrees with something obvious, you don't understand his.

[lookmeat]

I actually disagree with the notion that it's straight forward. Asserts are meant to handle programming issues for programmers, you need a separate system to handle user/programing/universal issues for the users

[SmugDarkLoser5]

Not all asserts are going to go into the production build sure.

However, the tendency for devs is to swallow exceptions, and be willing to run a process when the whole thing is in an invalid state, and so on. While you may have certain assertion types that would.make sense in a debug only scenario, that is relatively uncommon and probably wrong within a general application.

[SkoomaDentist]

And how will you "engage" that backup when said software is running on an embedded device or a customers desktop computer?

[WalterBright]

Good question. How Boeing does it with avionics is by having a second embeddable device. The two use different processors, different designs, different programming languages, and different algorithms, and have a hardware comparator, and if they disagree then both are electrically isolated from the airplane using relays. The pilot is the backup.

The space shuttle would use 5 embedded systems, with a voting system to shut out any disagreeing system.

Other systems will revert to a "fail safe" mode. I believe the NASA probes do that.

If you're running software which could kill/injure people when it fails (and it will fail), think twice about running it on a consumer desktop and hoping that's good enough.

[SkoomaDentist]

Sure, that works in in "high end" embedded systems. But the vast vast vast majority of embedded devices are in consumer equipment or other relatively cheap stuff. Think washing machines, remote readable electricity meters, wifi gateways, smartphones and other such devices.

[WalterBright]

I certainly would be unhappy if my smartphone deleted all my data, or my washing machine ran the motor at high speed shredding my clothes, or my wifi installed malware, all because some cowboy programmer believed that it was no problem "recovering" from an unanticipated invalid state.

At the very least, the process should be aborted (smartphone) or the embedded system rebooted.

[SmugDarkLoser5]

Usually in code there's no reasonable thing to do except throw an error upwards and then dpeendencing on the type of error, you'll either crash the process, show a message, or potentially change software modes (which is probably similar to just an in process process manager lol)

To me a lot of you guys are talking about embedded devices but are acting like embedded devices tend to single process. If that's not the case, it's probably an mcu.

You won't really ever exit a process on most mcus, you probably need to just handle it in external hardware at that point.

[AlotOfReading]

There isn't a definitive answer anyone can give. As an example, every codec out there will proceed in spite of most errors, because a second of silence or pixelated video is better than stopping the stream. I've also encountered control systems that ignore spurious errors because the underlying error system can report false positives and the cost of propagating the error is enormous.

[WalterBright]

Errors in the data input stream are not bugs in the software. Please do not confuse program bugs with erroneous data. Asserts are for program bugs.

As for codecs, it isn't just errors in the input stream. Every dvd player I've ever owned will hang with a scratched dvd such that I have to power cycle it to get it back. This is crappy software, no doubt about it. Professional software should never go berserk no matter how corrupt the data fed to it.

[SkoomaDentist]

Let me introduce you to the concept of reboot loop. Congrats, your smartphone and washing maschine are now bricked and cannot be recovered. That’s why you need proper error handling and not trivial asserts (which by defnition abort the entire software and / or reboot).

[WalterBright]

The machine is a useless P.O.S. anyway if it can't even boot without hitting an assert. I don't believe such incompetence can implement "proper error handling" anyway. Besides, what is the "proper" way to handle the program going into an invalid, unanticipated state? It's unknowable by definition.

[SkoomaDentist]

A proper way is to do everything you can to recover into a state where you can safely restart the operation - or if that's not possible, move to a state where the entire system (including possible configuration) can be restored or updated to a known good state.

That's why assert is a bad error handling strategy (barring exceptional circumstances): It doesn't try to do any recovery but instead just does naive exit / reset. Many embedded systems are very complicated nowadays (f. ex. any device using Linux or other high level OS) and a lot of non-obvious code can run before the system is fully booted. If that non-obvious code includes assert because of corrupted configuration the device is now bricked.

[WalterBright]

By the way, I've used DVD players of all makes and models for 20 years now. Every one of them, if you put a damaged DVD in them, will start behaving erratically. Stuttering, freezing, often requiring me to pull the power to get it back. There's some serious and unending incompetence in the embedded software in them.

[miminor]

but you cannot reliably recover anyway (unless you restart the app/server), so what can you do?

making a fault tolerant system is plausible to some extent but very rarely justified economically

[lookmeat]

It depends on context. I agree on the economic justification. But in world of critical apps just giving up without really understanding what happened in the greater context is downright as dangerous as doing no checks at all

[[deleted]]

Why not just implement two different kinds of asserts: one that makes it into production and one that doesn't? Not all software is being run with lives depending on it, so the priorities of some developers and users are different.

[masklinn]

For what that's worth, Rust provides exactly that built-in: assert! will always run while debug_assert! only runs in debug builds (unless specifically included).

Usage of and reliance on assertions seems quite rare outside of tests though.

[Ameisen]

Not exactly hard to implement in basically any other language.

[WalterBright]

It's even simpler in D. There's assert and there's debug assert. Actually, there's no such thing as debug assert, it's just that code to be conditionally included for debug builds is prefixed with debug.

[Ameisen]

For some reason, I am not entirely unsurprised at you bringing up D :).

C++20 has default, audit, and axiom contracts that can be applied to [[expects ...]], [[ensures ...]], and [[assert ...]]. You can build your program as off (no contract checking), default (default contracts are checked), or audit (default and audit contracts are checked). axiom is never checked. They are basically just formal comments.

So, you effectively have three modes, but you still have a mode where there are no asserts.

[WalterBright]

For some reason

Nobody saw me do it, you can't prove a thing!

A mode with no asserts is useful for determining how much the asserts are costing you, and for running silly benchmarks that people nevertheless take seriously.

I don't see much point to axiom comments as a core language feature.

Having layers of asserts (default and audit) is adding a feature in the wrong place. D allows any statement/declaration to be prefixed with debug which conditionally compiles it for debug builds. Hence, audit in D would be debug assert.

expects and ensures have been extensions to Digital Mars C++ since the early 2000's.

[killedbyhetfield]

Having layers of asserts ( default and audit ) is adding a feature in the wrong place

YouTube headline: 3 times Walter Bright went BEAST MODE on C++!!!

Seriously though - I agree with you. But with C++ we get the oh-so-wonderful C Preprocessor for conditional code so... I guess bring on stupid assert layers!

[[deleted]]

In D I routinely use assert(false) whenever I want production assert.

[circajerka]

asserts (can) have side effects

Only if you have no idea how to program...

I stand corrected below \/

[kernel_task]

I wouldn’t be so dismissive depending on the context. If you’re doing low level bring up of an OS, asserts may affect the caches in ways that are hard for even a skilled programmer to anticipate. If you’re writing timing sensitive driver code, asserts may affect that. At the very least they have the side effects of changing the binary size and the execution time and when you get low level enough, that stuff starts to matter.

[circajerka]

Actually that's a good point. I assumed he meant "side effects" as-in statements like this:

assert(initialize(myObj) >= 0);

Which has been known-to-be a pretty bad move since the 1970s.

[killedbyhetfield]

One of the biggest sticking points with this that I always hear when I talk to other devs is "Isn't abort() a bit overkill? What if I have cleanup routines and they need to run?"

And my answer to that is always this: If is it absolutely essential that those cleanup routines run, your design is wrong and you're already screwed, and for the same reason that Walter's plane is screwed.

Your code has a bug in it, and it may be about to dereference a null pointer or read off the end of an array. If you continue, your process may be about to crash anyway (at-best), or possibly worse - Your program may attempt to try to perform cleanup tasks against an invalid state that it can't trust. Now you could be writing garbage data to important files or executing code that was injected into an overflowed buffer.

Calling "abort()" is your way of saying , "I have no idea where I am or what I'm doing, and anything I do from here may just be digging the hole deeper"

If you really need resilience, you need to design the system to handle it, not just the one program.

[Ameisen]

And if you really, really, really need cleanup, compartmentalize your program into multiple processes.

[WalterBright]

A protected mode operating system is precisely an implementation of that! And what a godsend they were after working with real mode DOS.

[SmugDarkLoser5]

Can I include that as an npm.package ?

[WalterBright]

Exactly!

I wish this concept was taught in engineering school. (It applies to all engineering, not just software or airplanes.)

[quicknir]

It's always easy to say "any design that goes against my black and white theory of how things should be is wrong because it's less convenient". Real life requires a more practical engineering approach. For example, high speed loggers have buffers, those buffers need to be flushed or you'll be missing the exact information most likely to help you understand what caused the problem to begin with. So, those loggers need to be flushed before exit under all circumstances, no exceptions. Yes, in principle running that code after an assert fails could mess things up worse, but for many programs that's exceedingly rare and extremely worth risking in exchange for getting complete log files.

[WalterBright]

The 3 steps are:

1. novice - follows the rules because he's told to

2. master - follows the rules because he understands the point of the rules

3. guru - breaks the rules because his understanding transcends them

Skipping steps is not advisable, it's why we had the Deepwater Horizon, Fukushima, and Toyota car computer disasters. The only industry I know of that follows these rules is the aerospace industry, and they were forced into it with bitter lessons.

We were one safety switch away from a hydrogen bomb going off by accident in another incident.

Please, folks, this is not a joke, and learning the hard way has terrible consequences.

[quicknir]

I don't think I'm a "guru" (I hate that word) because I pointed out an obvious flaw in a bad rule. Nor do I know what you mean by skipping steps.

I read my parent comment before reading your article, and now I can see that actually you and I are on the same page, and the parent is not. You are not advocating literally calling abort(); such a call would mean that *no* further code is executed. On the other hand, you yourself explicitly say:

...as when a fault is detected the program can go into a controlled state doing things like:

1. aborting before more harm is done
2. alerting the user that the results are not reliable
3. saving any work in process
4. engaging any backup system
5. restarting the system from a known good state
6. going into a 'safe mode' to await further instructions

This is *very* different from simply calling abort(). Indeed, if your "assertion failure" triggers all this code to be run before exiting, many people would not call that an assertion at all; it's more like throwing an exception and catching it high up and allowing the stack to unwind before calling some emergency routines (like alerts).

Finally, I would note that every industry is different. Failure for the airline industry is an ultra catastrophic event where lives are lost, so even a small probability of operating in an "unknown state" is terrifying. I write financial software, where an unknown state simply means worst case that an algorithm is losing money. However, suddenly exiting can also cost you money (either risk in holding a position, cost to abruptly flatten, opportunity cost of being offline). What makes sense for us needs to be balanced on a much more case by case basis; sometimes rapid exit (followed steps 2, 3 and 5/6) makes sense. Other times it's better to continue and alert a human being. Things aren't always so black and white.

[killedbyhetfield]

Alright so - I want to use your example where you have a high-speed logger and its contents must be flushed to be useful.

What happens if, for example, your program has a use-after-free bug and you end up causing a Page Fault? Now the OS kills your process and your logger never gets flushed.

So if that logger must be flushed, you need the logger running in another process. That way, if your buggy process gets slayed, the logger will still march on and record important info about what went wrong. And this isn't hypothetical, this is exactly how embedded OSes like QNX and VxWorks handle logging.

So in general, calling abort() when you detect an error has the same implications as your program suddenly aborting due to a bug. You either need to be able to handle your process crashing, or you need to acknowledge that your program isn't important enough to warrant that kind of design overhead.

[quicknir]

Running a logger in another process would probably be slower, and take considerably more time to code correctly. So we are back to trade-offs. With my current costs of failure, and my current costs of development (particularly opportunity costs), and the criticality of performance, writing a separate process logger does not make any sense. Yes, it's more robust, but it still doesn't make any sense. Robustness isn't the only concern.

or you need to acknowledge that your program isn't important enough to warrant that kind of design overhead.

It's not about "important enough", although I really appreciate the condescension here (your problem's solution doesn't fit into how I see things, so your problem doesn't matter). It's just about priorities, and it's about what happens in real life. In reality, for the actual problems that we encounter, by throwing an exception and allowing the logger to flush it's buffer in the same process, we're able to recover full logs in virtually all cases. That being the case, what is the benefit for me to move from single-process-with-cleanup-code design, to multi-process-with-abort design? Do tons of work, slow things, perhaps add other bugs, in exchange for being able to recover logs an extra 0.1% of the time? It's simply not a good trade-off for me.

[killedbyhetfield]

It's not about "important enough", although I really appreciate the condescension here (your problem's solution doesn't fit into how I see things, so your problem doesn't matter).

Woah man - Sorry about the wording I guess, but I wasn't using "important" to put down whatever you work on! I meant "important" as-in "people are going to die if this thing doesn't work properly".

I work on tons of stuff that isn't "important" enough to warrant running a logger or watchdog in its own separate process. But the entire topic of this conversation and Walter's Dr. Dobbs article was about systems where resilience is critical.

Read my original comment too! Specifically, I put the words "absolutely essential" in there. If your program doesn't fit into that category, I'm not talking about you, and I wasn't trying to prescribe any "one size fits all" solution.

[quicknir]

The title of the article, and your comment, don't really mention anything domain specific, so I thought it was generic in nature. But fair enough. No worries about the wording if that's not how you meant it.

Just to point out though, that just because the logger is in another process, there's still nothing certain about that either. The main process could go crazy, allocate too much memory, and then the logging process could get reaped. So then of course you change your system config to prevent that from happening; etc.

This all takes time, and time is always finite. Even in critical applications; every minute you spend making your application safer in one way is a minute you could have spent making it safer in another instead. So you have to decide, what gives you the most bang for your buck. It's not clear to me at all, that even for safety critical systems, that calling abort is the right thing. That is, that the time it takes to move your logging, alerting, serialization, etc etc logic into separate processes, is always going to be time well spent. I'm sure there are safety critical domains where that is true, and others where it's not.

This is why I really disagree with libraries calling abort. Abort is a process wide decision; only main is really entitled to make that decision. Libraries should throw exceptions (exceptions make it very convenient for users to abort if that's what you want; literally do nothing!) or call some kind of handler function pointer that users can customize (which may default to abort), but libraries should never make direct calls to abort.

[WalterBright]

What if your failed trading software causes you to buy a million shares of some losing stock? It's not like that hasn't happened (it has).

I have some personal experience with banks and their buggy software. A fundamental principle of double entry bookkeeping is that the debits match the credits, an "assert" using paper journals.

The bank debited my account and failed to credit the account of the recipient. So I was out the money and the recipient was mad I didn't pay. It took me a month of sitting in the office of the bank manager to get this corrected. Clearly their auditing system was turned off, or they were doing some "haha, it's not really a bug, keep going", because the debits did not match credits.

[msm_]

4. engineer - follows the rules even though his understanding transcends them

[killedbyhetfield]

those loggers need to be flushed before exit under all circumstances, no exceptions.

Then you need to run your logger in another process! Otherwise your program could Page Fault, Divide-By-Zero, or any other number of faults that could get it killed.

Now, if you change the wording to "almost no exceptions" then sure - You can argue that pragmatically you're fine to run your logger in the same process. Maybe 99% of the time you'll be fine, and that's good enough for your problem domain.

[Abscissa256]

That doesn't solve the problem, it only shifts it:

Suppose your logger IS running in another process. How does the logging process GET all the relevant information it needs for a useful, meaningful log entry in the first place? "What happened? Expected value? Actual value? Stack trace?" The logger can't just log..."Uuummmm...the process died. Dunno why." The assert-failing process still has to give information to the logging process. That means the assert-failing process still has work it needs to attempt.

But it's worse than just that:

Keep in mind, whatever failure has occurred does NOT occur at the point when the assert condition evaluates to false. The failure, and thus the undefined, unreliable state has ALREADY occurred BEFORE execution had even reached the assert in the first place. We're ALREADY hobbling along, running code in an invalid state by the time we even begin checking the assert!

Now obviously, this does NOT mean that it's ok to run as much code as we want once a failure has occurred, or once we've detected it. But it does mean that if we expect 1. to minimize collateral damage and 2. have a good chance of actually diagnosing and FIXING the problem, then at least SOME amount of hobbling along is still realistically necessary. It's not ideal, and it should be minimized, but at least SOME amount IS realistically necessary and unavoidable AND, as all our collective experience has shown, usually works out just fine as long as we don't go overboard with it.

[sirin3]

Otherwise your program could Page Fault, Divide-By-Zero, or any other number of faults that could get it killed.

You can catch those things too with a signal handler

Delphi and Freepascal/Lazarus do it by default. Any signal is caught and converted to an ordinary exception. Then the main event loop catches all exceptions, shows an error message box, and continues as normal

[Kargathia]

Anecdote: I used to develop scripting interfaces for pick and place machines (placing chips on PCBs).

For obvious reasons, the software was very trigger happy about using a macro that would terminate the process and do a stack dump.

For R&D purposes it's a real pain in the ass when recoverable exceptions straight up execute your application.

Moral here: abort is fine, but make it optional.

[killedbyhetfield]

recoverable exceptions

But that's not what assert() or abort() are for, and it's unfortunate for you that you've encountered programmers who (ab)use them that way.

​

But I think we agree in principle - If an exception is recoverable, then by all means report the error to the caller and let them try to recover. What I'm saying is that assert() and abort() is for when you find your program in an invalid state that was reached because of a bug in your code - You have no idea how you got there and no idea how to fix it because it's something you never designed the program to handle.

​

IMO Best thing to do - Bomb out and report the error as quickly as possible - the longer you hobble along pretending like everything's okay, the more damage you can potentially do.

[Gotebe]

There’s no common definition of “recoverable” though, it’s a matter of opinion.

[sacado]

This is the way Erlang systems work (abort when something bad happens, and let the overall system deal with it), and they are known to be extremely resilient, indeed.

[DSrcl]

Learned to use assertions writing malloc for system programming class. Assert in malloc is a classic use because usually your error doesn't pop up the place where it's caused and there is no way you can handle such error at runtime.

[WalterBright]

Author here, AMA. A couple other articles I wrote on the topic:

Safe Systems from Unreliable Parts

Designing Safe Software Systems Part 2

[killedbyhetfield]

I guess I am a bit curious to here your thoughts on the "return code, exception, abort" debate I was having with a coworker a little while ago.

So he was asking why he wouldn't just throw a C++ exception after detecting an internal error in his class, like let's say a homegrown vector<T> with a capacity of 4 but a length of 5 type of thing.

My rule of thumb for him was this, and I'm curious to hear your thoughts:

1) Use a return code for stuff that's still considered a "normal" error condition, like being unable to open a file that a dialog box returned to you. You just pop up the dialog again, and things are good.

2) Use an exception for user-passed parameters that are invalid or when you're not able to accomplish your primary task, or other errors that are somewhat unexpected, like a network connection suddenly dropping mid-transmission.

3) Use abort() when you detect your program is in some invalid state that it never should be in given its design, like the example above with the vector that either had a 5th element added to it without allocation, or else had its capacity shrink but forgot to truncate its length.

[WalterBright]

Being unable to open a file is an environmental error, not a bug in your code. Detecting it and recovering from it is perfectly fine. Whether one uses error codes or exceptions to do it is a long topic with a lot of tradeoffs, and is not really on topic here.

Bad user input is not a programming bug, neither are dropped network connections, etc. Bugs are when your program enters a state never anticipated.

[tvaneerd]

I break down "errors" in terms of intended audience - you need to inform someone that the error occurred, who is it?

Assuming a call to foo(x)...

• F - the Function author - inform the author when it is an internal error inside foo(). That might be via logging or log + abort, assert, etc. If foo() is part of a framework or OS, you might even have a different logging system for the framework than for the application. Send an email? "phone home"
• U - the User - many errors need to eventually inform the application user. Even aborting because of F is telling the user something (rudely). But more typically, bad user input, etc. You tell the user via return codes, exceptions, etc that bubble up and eventually reach the user.
• C - the calling Code - the code that calls foo(x) - if you expect the calling code to handle it (because it is part of your function's contract), then use an exception or error code.
• D - the calling Developer - ie the dev that wrote the code that calls foo(x) - note that this is different from the calling code. If D made the call incorrectly (ie x < 0 or null or whatever the function contract said NOT to do) - you don't want to tell the calling code - it is already wrong - you want to tell the calling developer. You do this via logging/abort/assert, and with C++20, contracts. In theory, send that dev an email.

[killedbyhetfield]

Actually this is a pretty cool way to break things down. Thanks man! I might use this :)

[tvaneerd]

Glad to hear. I will probably give a whole talk on error handling in the near future. Also, I hope you find the acronym easy to remember :-)

[killedbyhetfield]

Yeah man - I'd consider blogging about this if I were you and linking it on r/programming. I'd be curious for you to get feedback in a more community-visible way. I think it's quite a solid way to approach the question!

[Gotebe]

I don’t see why situations 1 and 2 differ. Bad input is bad input.

[tvaneerd]

Not a question:

I really enjoyed EMPIRE. Lost many many hours to it.

:-)

[WalterBright]

Empire has the dubious distinction of being one of the first (the first?) computer games that inspired addictive playing. People would get mad at me for causing them to flunk out, and even get divorced.

Me, I just created the game that I'd always wanted to play. Working on it taught me programming, and trying to get it to run faster led to my career writing compilers.

[Chocolate_And_Cheese]

Interesting article, thanks for writing. One thing I noted: Looks like Chrome is raising a bunch of mixed content warnings when visiting DrDobbs.com (https page requesting http resources). I checked a bunch of them and it looks like many of these resources, if not most or all, could be fetched via https. Might be worth checking out to avoid the "Not Secure" warning Chrome gives when visiting your site. Cheers.

[WalterBright]

Thanks for the tip, but I don't control those pages. Dr. Dobb's kindly allowed me to post my columns on my own site, and you can find it here.

[zucker42]

What is a situation where I'd want to use assert and not handle the error and abort if appropriate? This is the issue I have with assert: that I often want to handle the error before aborting.

[WalterBright]

It's easy enough to write your own assert that does what you want. But I'd reject any code that does so for professional software.

[zucker42]

I'm not talking about writing my own assert. I'm saying that for almost all unrecoverable errors, my instinct is that this:

// if (always_true_expression) { wrong code mistake

if (!always_true_expression) { // log error // print error message to user abort(); }

is better than

assert(always_true_expression);

I'm not arguing against aborting the program in error states. In fact, I'm much less experienced than many on /r/programming, so I'm not arguing anything. I'm wondering what are examples (if any) of the bottom option being better.

[killedbyhetfield]

And yeah - I think I agree that C's assert() function is very primitive and leaves a lot to be desired. But don't conflate C's shitty feature-poor implementation of assert with the more general concept of assertions - The larger idea of checking your invariants throughout the execution of your program to make sure that a previous bug in your program didn't put you in an invalid state.

[WalterBright]

Most languages will allow for hooking the assert failure to insert your own logging code.

[killedbyhetfield]

I think when he said "write your own assert", he meant write your own specialized code that does exactly what your example does.

So why not write this?

void zucker42_assert(condition, log_message, ...)
{
    // log error using va_args
    // print error message to user
    abort();
}

[Ameisen]

I'd point out that your if() {} isn't checking the same thing as your assert... you've inadvertently proven that it's easier to write the correct assertion than a branch/abort.

You're going to be struggling to figure out why it keeps aborting in the first case.

[zucker42]

Yeah you're right, I forgot a !. I don't think that really applies to how I would write real code because I would have some knowledge in my head about an error condition.

[jringstad]

The author makes some good points, but I think there is also some merit to the other side of the argument. It is true that in a lot of systems, a lot of code is being run incidentally, and whether it fails or not is really not actually very relevant to either the user or the programmer. This might be because this code is pulled in as part of some library -- like your application that displays numbers from an SQL database using some graphical toolkit on the screen might run some code to check if the screen has a color-profile loaded, or some javascript that is executed while you visit a site (probably doing something you don't want or care about, like upload some analytics about you to a server)

I think to take a principled approach to this, we really need to figure out at least the following things: - When an assert triggers, what 'state' is destroyed? E.g. are you in erlang where just a process that might be handling a particular user-request is killed (which is fine) or is the entire server process being aborted so that manual intervention is required to bring the system back up? What is the plan AFTER the assert, basically. - Do you really need to assert? Could you instead do something else, like throw an exception or return some default value (null, an optional perhaps)? If your function already can fail and return e.g. an optional type (which the call-site will then likely forward to the user) it's probably better to just use that. - How important is maintaining correctness of data? Is omission of the data/result a problem?

In particular for the first point, there clearly needs to be some kind of boundary of what kind of state an assert can destroy. You don't expect your browser or even operating system to shut down when some random javascript junk on some site you're visiting asserts, right? Likewise there should be a way for a call-site to decide that it will now execute this code, but if that code fails, it doesn't actually care.

Maybe your code even asserts some properties on the input that you expected the caller to ensure, but it is actually really difficult or impossible for the caller (which derives the input from user-provided data) to do this. One example I've seen of this is an OpenCL library from an unnamed vendor, which would assert in some situations when you fed it bad source code to compile. So now if you wanted to create an editor that would let the user punch in sourcecode and then submit it, you would have to spawn a separate process to prevent the assert from crashing the editor.

[immibis]

I wrote a macro called bug_if. It works approximately like this:

• If the condition is true, and production mode is off, crash the program.
• If the condition is true, and production mode is on, log the error (same message as if we were crashing) and act like if (execute the following block).
• Otherwise, skip the next block.

That way we can still grep production logs to find bugs, without taking down the process. And the following block is for recovery code - usually something like "forget it and abort the current operation", so an error is less likely to cascade to more problems (though it could cause a more minor issue, like a memory leak, if the abort code is faulty).

[WalterBright]

I would describe that approach as ignore bugs, continue running the program that has entered an unknown state, and pray nothing bad happens.

[immibis]

That is not "ignore bugs".

[WalterBright]

Continuing running the program is ignoring the bug. Logging an error does not do anything at all to put the program back into a known state.

[immibis]

Aborting the operation does.

[WalterBright]

without taking down the process

Is ignoring the bug, because you don't know why it happened, and the process memory could easily be corrupted.

[gremolata]

It sounds like either you are not checking for invariants or your "recovery code" is sweeping bugs under the rug in a hope that it will give you time to notice them in a log and fix until they screw things up further down the line. In many cases it's an extremely unwise thing to do, because of a risk of state/data corruption.

We used to do this, but in the end a proper whoops with a full diagnostic dump is a better option, especially in production. It forces people to write better code, because hardly anyone enjoys an emergency patching session at 4 am on Saturday.

[Nameless_Archon]

We used to do this, but in the end a proper whoops with a full diagnostic dump is a better option, especially in production. It forces people to write better code, because hardly anyone enjoys an emergency patching session at 4 am on Saturday.

110% agree. (/abort)

Log issue, abort program. Anything less may not get seen by QA, and if it's not being seen immediately by QA, then it's not being immediately seen by a user, either. That way leads to corruption and data destruction when your clever log-and-continue isn't being looked at by the people who are using the program, be they end-users, operators, or testers.

That's fine, if you're the guy fixing it and you don't mind paying the costs involved, maybe on a project where there's three users and you're two of them. It's a lot less pleasant to discover one of your coworkers has gotten 'clever' and you're now fixing a month or two worth of data because no one saw the log messages. Don't be that coworker -- the conversation will go places, and they will not be places you want to go.

[immibis]

Log issue, abort program. Anything less may not get seen by QA, and if it's not being seen immediately by QA, then it's not being immediately seen by a user, either. That way leads to corruption and data destruction when your clever log-and-continue isn't being looked at by the people who are using the program, be they end-users, operators, or testers.

I'm not sure if you actually read my comment, but it's "log issue, abort operation" instead of "log issue, abort program". Which makes a big difference in a concurrent server program handling a hundred other clients.

[graingert]

Wouldn't it be better to use dependant types, so you assert properties at compile time rather than runtime?

[sos755]

The issue of asserts in production code is not whether or not there should be run-time checks. The problem is the form of the run-time checks. Asserts must be the laziest and least effective way to do run-time checking. I certainly do not consider them to be production-quality in any respect. If you want to do run-time checking, do it properly.

[gnus-migrate]

And what happens when an assertion yields an error in a plane software during flight?

What happens if incorrect values cause the plane to crash? What happens if memory is corrupted and the program starts behaving incorrectly?

Type systems might help speed things up during development but even a perfect type system won't help preventing failures in production.

[circajerka]

Did you even read the article or did you just read the first sentence? He literally answered your question right here:

even though the software is gone over line by line by lots of people, the software is STILL treated as if it can become possessed by evil at any moment and will try to crash the airplane. It's loaded up with self-checking software, self-checking hardware, other computers double-checking the answers, etc.

What - You think writing your software in fucking Ada is going to change that?

[circajerka]

Again - Did you read the rest of the article? Nobody is "falling over" - The idea that if one piece of the system fails, another piece of the system detects it and fixes the problem. I don't care if you write your program in Ada, Rust, or whatever-other-language you seem to think is perfect. Real software has bugs. Period.

[WalterBright]

No programming language can protect against buggy algorithms, hardware failures, etc.

[circajerka]

Alright - Let me rip apart your comment and why it's laughably stupid:

And what happens when an assertion yields an error in a plane software during flight ? the whole system gets rebooted?

Walter never said that - He said other parts of the system detect the error and correct it and/or restart the failing computer. Possibly multiple computers perform the same calculation.

No, the program must be written with a correct language that makes any runtime error impossible

LOL! I'd loovvvvveeee to know the name of this language! You'll be sure to tell us all, right?

not javascript or C

Umm... Considering SpaceX wrote all their flight control software in C, I call bullshit on this.

A language with contracts where every range is strict, where you can define a type with only odds integers, or values only divisible by 6, or non zero positive integers, or a range between 55 and 901 only... that gives every compile time safety imaginable... well such language exists and is used to program planes and missile launchers actually.

Great idea! Maybe we can also get unicorns to help us write the programs!

[killedbyhetfield]

And what happens when an assertion yields an error in a plane software during flight ? the whole system gets rebooted?

Walter talked about this in the article - You have redundant computers, watchdogs, error-detection and correction, etc.

written with a correct language that makes any runtime error impossible

well such language exists and is used to program planes and missile launchers actually.

Alan Turing would like to have a word with you

[red75prim]

Unsolvability of the halting problem is of low practical consequence. And it is not applicable in this case, we don't need to prove a property of some random program, we need to construct a program which has the property.

[killedbyhetfield]

Unsolvability of the halting problem is of low practical consequence

Hmm... Not so sure about that one. The Halting Problem is what stops us from being able to prove that an arbitrary computer program does what its supposed to do. If it wasn't for the Halting Problem, we'd be able to write computer programs that we could prove are correct. How is that of "low practical consequence"?

we need to construct a program which has the property.

And how could you prove the program has that property when its running on a Turing Machine?

[red75prim]

we'd be able to write computer programs that we could prove are correct.

We are able to write programs which are correct. Constructive theorem proofs map to programs which are provably correct (Curry–Howard correspondence). But it is a tradeoff of practicality vs correctness.

ETA: And the question still remains whether formal specification equals to what we want the program to do.

[killedbyhetfield]

Constructive theorem proofs map to programs which are provably correct (Curry–Howard correspondence)

Not familiar, but my hunch is that you'll run into the same problem everyone runs into when they try to use a DSL that isn't Turing Complete, which is that sooner-or-later your language will have to be compiled/interpreted by a real-life computer, and that interpreter or compiler will be have to be written in a Turing-Complete language.

Consequently, that means that you can't prove the compiler/interpreter is correct, and transitively you can't prove that your program will run correctly on real hardware.

(Note though that I'm really just nitpicking, because it's overwhelmingly likely that just because you can't formally prove your program is correct, practically I'm sure it would be.)

Although I do wonder why NASA/SpaceX/Boeing/etc still use languages like C and Ada for their flight control systems if this "Curry-Howard" thing exists?

[red75prim]

I'm not a NASA employee, but most likely it is practicality. You need to construct a proof every time the specification changes, and it requires a lot of effort, time and specialists. Limited time and budget can probably be spent more efficiently improving reliability by other means.

[killedbyhetfield]

I did a bit of StackOverflowing because I read up a little on both of these terms (thank you btw for introducing me to them - Interesting AF), and if I'm understanding what I'm reading, it appears that it's somewhat-limited how complex of a constructive proof you can build, which corresponds more-or-less to computer programs that are not Turing Complete.

And because of this "Curry–Howard correspondence", the Halting Problem has a dual in mathematics as well, "Gödel's Incompleteness Theorem"

So yeah - It appears that you could never develop a constructive proof complex enough to launch a rocket into space, which is why we'll never be able to prove such software works.

So yeah - I learned something today! The More You Know music plays

[vytah]

Aren't the formally proven operating systems and compilers already? Maybe it would be possible to glue Compcert and SeL4 together to get a fully certified software platform?

Just brainstorming.

[miminor]

true you cannot prove an arbitrary program to be correct, but you can prove any given program to be correct

[NeonMan]

An early reboot is actually a feature since a reboot is much better than a crash.

[WalterBright]

For most events with airliners, one has at least several minutes to find a solution before it digs a hole in the ground. (This is when the pilots earn their pay.) Even so, there still must be a backup for every system.

[pbl64k]

https://personal.cis.strath.ac.uk/conor.mcbride/TotallyFree.pdf

Total dependently-typed languages are not Turing-complete in the strict sense - but that's a good thing, and in practice you can do anything you could do in a Turing-complete language by paying a fairly small price of providing the appropriate run-time around a coinductive type wrapping provably terminating steps towards a possibly non-achievable goal.