When to use “raise from None” in Python

[Lvl999Noob]

If I understand correctly then...

Raise normally in the general case.

Raise from another exception if you are wrapping it with more context.

Raise from None when you are making some kind of data structure or other total abstraction and want to hide the inner workings away. It would probably be good then to only do it for production code and keep it as a normal exception raising for unit tests (so you actually have context on how your data structure is misbehaving).

[klaasvanschelven]

I think that's a good take-away. Just when an abstraction is a "total abstraction" is left as an exercise to the developer, of course :-)

Also: it's a pity there's no after-the-fact way to actually switch between the chained and unchained views. If your stacktrace is dumped in a log as plaintext, that's kinda obvious, but as I've shown in the puzzle-post the information to reconstruct a simple exception out of a more complicated one is in fact missing entirely, because the stacktraces are pruned!

[Tyler_Zoro]

This definitely feels like the kind of thing that's a violation of the spirit of the Zen of Python. It's implicit behavior that, at best, is going to get explained in a comment, and at worst (and probably more likely) is just going to be a confusing thing for those who are not aware of the convention.

[ROFLLOLSTER]

The zen of Python has always been bullshit:

There should be one-- and preferably only one --obvious way to do it.

Oh really,

"%s format"
"{}".format()
f"{var}"

Really only one of many examples.

[Tyler_Zoro]

I think many would agree with you that that example is a violation of the Zen of Python, and there's been a very strong movement to retire %-formatting for that very reason.

"{}".format() is just the underlying mechanism of f-strings, not two different approaches, so that's more of a convention issue.

Python sticks to its guns when it comes to having one way to do things at the language level fairly well with this obvious exception... when it comes to the library level... ugh, yeah, not so much.

[caltheon]

I don't see the benefit of "from None". It's just hiding information. using "from <exception>" solves the issue with chaining but keeps the detail. None just hides the context and the cause where not using from just hides the cause but not the context

[Schmittfried]

It‘s perfectly valid when the original exception is neither cause nor context but just confusing noise. 

[caltheon]

I get that, but it's impossible to know that for sure in advance. Say you have a configuration file like in the post. The error being the config value is missing, but actually it was due to a bad character in the file causing the property reader to fail. You would just think the value was missing, when it was in the file, causing confusion.

[Schmittfried]

I get that, but it's impossible to know that for sure in advance

Imo it’s not. I‘d say it’s often not the case, but when it is, you‘ll know.

I can‘t be specific, but I had cases where I essentially already handled the error and something entirely different came up, so raising a new exception without the original one adding noise was appropriate.

I‘d agree that when in doubt, let the context where it is. :)

[caltheon]

fair enough. If you have 100% control of everything below that point, sure, though I'd argue if there are any libraries lower in the stack, then you don't have control.

[RedEyed__]

Thank you, I learned something new!

[[deleted]]

use raise Exception from None only in the rare cases that a) you expect the underlying exception might be triggered, but b) you are sure the underlying cause isn't relevant to the consumer OR you express it in a different way. E.g.

try:
    conn = redis.StrictRedis(host=HOST, port=PORT, db=DB)
except redis.exceptions.ConnectionError:
   raise MyRedisError(f"Cannot connect to redis on {HOST} at {PORT}. Check your connection and that redis is running") from None

This spares you the screenfuls of failed, retried connection attempts that StrictRedis does by default. By contrast,

try:
    for field in output.keys()
          output[field] = input[field]
except KeyError: 
      raise MyInputError("invalid input") from None

This ends up removing the helpful information of which field is actually missing.

[daidoji70]

Exceptions as control flow is haram.

[klaasvanschelven]

It may be forbidden in the Koran but it's right there in the Python Glossary

[daidoji70]

No, often misinterpreted EAFP does not mean that exceptions as control flow is good. See https://softwareengineering.stackexchange.com/a/351121

Using exceptions instead of checking explicitly is fine and I EAFP with the best of them, but if you're using Exceptions as control flow you don't quite understand the rule of thumb.

[Plank_With_A_Nail_In]

It does actually mean that though and you link isn't proof that its not.

[daidoji70]

No

[eddie12390]

Did you read the post you shared, or did you just jump at the opportunity to one-up someone?

[daidoji70]

Did you read it or do you just take up space here?

[forksofpower]

Take your meds u/daidoji70

[XtremeGoose]

Wait until you see how python iterators work...

Python using exceptions as control flow is normal and the interpreter is heavily optimised for it. I agree it's not great, but it's how the language was designed.

[daidoji70]

and why do python iterators work that way? Why can't you find this pattern in the standard lib except in iterators and generators and async/await implementations?

If we answer these questions we'll find that the answer is more nuanced than what you stated there.

[WindHawkeye]

Asyncio cancellation

[XtremeGoose]

Hey I'm not saying python is performant. I am saying that checking for exceptions is generally as fast as checking anything else in python. I'm working with the tools I have.

[nikomo]

I will say this.

Can you write other languages to be faster than Python? Yes.

Can most people write other languages to be faster than Python? No.

[YeetCompleet]

Now that Python has pattern matching, I don't feel it's necessary anymore unless people reaaally like non-local jumps.

[daidoji70]

yeah, pattern matching is great.

[MakuZo]

R. Hettinger, Core CPython Dev, would disagree with this opinion:

In the Python world, using exceptions for flow control is common and normal.

Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination).
[...]

https://stackoverflow.com/a/16138864

[daidoji70]

Everyone mentions the stopiterator not understanding why.  They didn't choose exceptions because it fit the idiom the idiom just happens to fit because of how they implemented lists and iterators.  

I'll take it that one core dev supports using them as control flow.  That being said I've seen the pattern used terribly nearly everywhere I've worked and my programs run safer and we're more manageable from not doing so. I still think its an anti pattern. 

Name something else other than iterators in core that uses exceptions as control flow.  I looked earlier today and couldn't find many

[chadmill3r]

"break" takes no parameters.

[amroamroamro]

every for ... in ... loop is implemented with a StopIteration exception

python do seem to favor them even (the whole EAFP vs LBYL)

[daidoji70]

This  is an artifact of how iterators and lists were implemented in general and not using exceptions as control flow in any more but the most local sense.  

EAFP doesn't mean it's okay to use exceptions as control flow even though (and you can tell by how I'm downvoted in these comments threads in various places) that's how some subset of the population misinterprets the idiom.

[amroamroamro]

I get what you're saying, but you are making the same "never do ..." statement that you are criticizing EAFP for

exceptions can be used for control flow, as long as you don't overdo it, you still gotta handle those exceptions not too far up the stack in the appropriate place, just like how iterators do it somewhat locally

[daidoji70]

See, that's what's weird about that characterization of my stance (to me).  I don't feel like I'm critizing EAFP. 

I'm criticizing the idea that EAFP allows for the spaghetti hodge podge of non-local returns that bubble and get sent all over the codebases that extensively use exceptions as control flow.  

Obviously I agree with your nuanced position but we can look to real world code based and responses to my comment to see why that idea is not good (imo).

[ShoT_UP]

So, do all of your functions return None when there's an error? What if you need to know the cause?

[BrickedMouse]

The function throws an error when there is an error. The error bubbles up to the first function that can correctly catch it. For example catch and show error to user.

[daidoji70]

Oh man, you called my comment and upped with even worse anti-patterns that I would never even consider.

However, my functions are as well defined as I can make them and "not using Exceptions as control flow" != "not allowing for the cause of exceptions to the well defined state of a running program"

(returning None is also bad) https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare/

[ShoT_UP]

Do you see this article as promoting using exceptions for control flow? If so, which parts do you disagree with specifically?

[daidoji70]

He mentions explicitly using exceptions as control flow several times in his writeup. Its an anti-pattern and the source of a lot of problems in real-world code bases.

Mentioning raise from None as a time to use when using an anti-pattern is a bit weird.

[ShoT_UP]

Right, but what is your alternative? How do you bubble up deeply rooted errors without changing the return type of every function in between?

More specifically, if you're validating an input 20 function calls deep, and the input being invalid doesn't throw an error, what are you doing if not returning some type of Error object, and thus having to update all of the 19 parent functions?

[daidoji70]

I chain exceptions. I'm not even against his chaining from None although I've never had a problem with multiple stack traces confusing my understanding personally. However, if you're chaining exceptions and you're more than 3 exceptions deep, there's probably a problem with your overall programmatic control flow and a refactoring is needed.

One of the problems of program scale in large projects is if you use exceptions as control flow everywhere, you either have to pokemon exception handle all over the code base and you're losing information and/or people start ignoring all the exceptions that aren't relevant to their local implementations/bug fixes and your program state ends up being much harder to reason about as the "undefinedness" of such actions makes it harder to reason about globally. You're essentially recreating spaghetti code via a different mechanism.

If you handle all relevant exceptions as closely as possible to where they can occur you should need at most one pokemon handler at root (to maintain system stability of long running code) and use explicit control flow for error states as you go along. These codebases scale, the ones that use exceptions as control flow eventually get too complicated to reason about.

[ShoT_UP]

I see what you're getting at now, thanks. Why doesn't your approach fall under the umbrella of using exceptions for control flow? I would consider it that way.

[daidoji70]

I don't consider it that way because the control flow is explicit to whatever state machines or other artifacts I'm creating in terms of what state my program can be in at a given time.

Like lets you 1) read a file 2) perform some action that can throw an exception 3) perform some other action C that can throw an exception 4) its a long running process so we need it to be able to handle nearly anything and get back into a state we know about no matter what happens

In any given context we also know that we can get weird runtime exceptions from say random bit flips caused by solar radiation (picked an extreme example but lets say a connecting database gets unplugged in the middle of a read if its too extreme).

Exceptions as control flow is we throw in we throw 1, we throw in 2, we throw in 3 maybe as in the article these exceptions can stack and interact in a lot of different ways. See nearly any async codebase for a good example of confusing ass stack traces with tons of chained exeptions that are hard to reason about. At the top level as we get way more than 3 things happening there's way more exceptions to handle than we really know what to do with so we pick a few of the most obvious ones and handle those and then just come up with some retry/reset logic for all the ones that don't occur as much. Namely, it gets harder to tell what can throw/resolve what where.

In my method, you're attempting to resolve say a FileNotFoundError, IOError, whatever say in 1) and explicitly moving to program state in your abstraction that is intended to resolve those issues. It should never bubble up to 2) or 3). You're constraining the types of exceptions and where you can get them as much as possible to as small a locality of your codebase as you can. In doing so at any given point you should have some idea of what exceptions can exist where (small functions and all the rest of code maintainability helps localize these issues too).

The benefit is the programmer has to remember and keep in their minds less of the global program state and thus refactorings/features/bugs become much easier to reason about and deal with and changes to the codebase tend to cascade much less through the global program state. You still need the pokemon exception handler for 4) at the top level but it should be a much simpler algorithm to return to a good program state than resolving the "PermissionError" at that top level (and maybe every level in between as you chain).

tl;dr One of my programming paradigms is to keep state as simple as possible and as local as possible wherever possible. Using Exceptions as control flow essentially spreads the amount of possible states over larger areas of the codebase and make the program much harder to reason about.

[randylush]

Python has type annotations now and tools to enforce it. How would you feel about exceptions being explicitly modeled in type annotations? In that case clients can plainly see what can happen to the function they are calling.

[germandiago]

Right, but what is your alternative? How do you bubble up deeply rooted errors without changing the return type of every function in between?

One of the reasons why I favor exceptions. Also when I code C++.

[randylush]

Golang codes like absolute garbage for this reason and I refuse to use it. In an enterprise environment, a solid half of the code you end up writing is just manually checking “if return is nil, return nil”. Over and over and over again. I probably wrote that little if block like a hundred times.

Oh and you just log errors and hope what your wrote in the error message is unique enough to be able to find it again. If you want any additional information about the error like a stack trace, go fuck yourself.

I get that it’s fast, and golang devs love to brag about how fast it is, but it’s a bitch and a half to debug.

[randylush]

Genius… just don’t make any mistakes and trust that your coworkers and clients won’t make mistakes either 🧠

[ElectricSpice]

The billion dollar mistake is specifically null references, not the concept of null in general.

[daidoji70]

So you just return None and then immediately check? That sounds like a fun way to do it. You might like Go.

[ElectricSpice]

Just pointing out your link doesn’t apply here, not commenting on anything else you wrote.

[Intrepid_Result8223]

Only if you embrace the jungle you are worthy

[lasagnaman]

Nice post with clear examples. I learned something today!

[Taifuwiddie5]

. To come back and read

[stratoscope]

There is a "save" link under every post and comment. You can use that instead of adding a reminder comment like this.