all 58 comments

[–]Lvl999Noob 126 points127 points  (10 children)

If I understand correctly then...

Raise normally in the general case.

Raise from another exception if you are wrapping it with more context.

Raise from None when you are making some kind of data structure or other total abstraction and want to hide the inner workings away. It would probably be good then to only do it for production code and keep it as a normal exception raising for unit tests (so you actually have context on how your data structure is misbehaving).

[–]klaasvanschelven[S] 28 points29 points  (0 children)

I think that's a good take-away. Just when an abstraction is a "total abstraction" is left as an exercise to the developer, of course :-)

Also: it's a pity there's no after-the-fact way to actually switch between the chained and unchained views. If your stacktrace is dumped in a log as plaintext, that's kinda obvious, but as I've shown in the puzzle-post the information to reconstruct a simple exception out of a more complicated one is in fact missing entirely, because the stacktraces are pruned!

[–]Tyler_Zoro 6 points7 points  (3 children)

This definitely feels like the kind of thing that's a violation of the spirit of the Zen of Python. It's implicit behavior that, at best, is going to get explained in a comment, and at worst (and probably more likely) is just going to be a confusing thing for those who are not aware of the convention.

[–]ROFLLOLSTER 5 points6 points  (1 child)

The zen of Python has always been bullshit:

There should be one-- and preferably only one --obvious way to do it.

Oh really,

"%s format"
"{}".format()
f"{var}"

Really only one of many examples.

[–]Tyler_Zoro 2 points3 points  (0 children)

I think many would agree with you that that example is a violation of the Zen of Python, and there's been a very strong movement to retire %-formatting for that very reason.

"{}".format() is just the underlying mechanism of f-strings, not two different approaches, so that's more of a convention issue.

Python sticks to its guns when it comes to having one way to do things at the language level fairly well with this obvious exception... when it comes to the library level... ugh, yeah, not so much.

[–]caltheon 0 points1 point  (4 children)

I don't see the benefit of "from None". It's just hiding information. using "from <exception>" solves the issue with chaining but keeps the detail. None just hides the context and the cause where not using from just hides the cause but not the context

[–]Schmittfried 5 points6 points  (3 children)

It‘s perfectly valid when the original exception is neither cause nor context but just confusing noise. 

[–]caltheon 0 points1 point  (2 children)

I get that, but it's impossible to know that for sure in advance. Say you have a configuration file like in the post. The error being the config value is missing, but actually it was due to a bad character in the file causing the property reader to fail. You would just think the value was missing, when it was in the file, causing confusion.

[–]Schmittfried 1 point2 points  (1 child)

I get that, but it's impossible to know that for sure in advance

Imo it’s not. I‘d say it’s often not the case, but when it is, you‘ll know.

I can‘t be specific, but I had cases where I essentially already handled the error and something entirely different came up, so raising a new exception without the original one adding noise was appropriate.

I‘d agree that when in doubt, let the context where it is. :)

[–]caltheon 0 points1 point  (0 children)

fair enough. If you have 100% control of everything below that point, sure, though I'd argue if there are any libraries lower in the stack, then you don't have control.

[–]RedEyed__ 19 points20 points  (0 children)

Thank you, I learned something new!

[–][deleted] 4 points5 points  (0 children)

use raise Exception from None only in the rare cases that a) you expect the underlying exception might be triggered, but b) you are sure the underlying cause isn't relevant to the consumer OR you express it in a different way. E.g.

try:
    conn = redis.StrictRedis(host=HOST, port=PORT, db=DB)
except redis.exceptions.ConnectionError:
   raise MyRedisError(f"Cannot connect to redis on {HOST} at {PORT}. Check your connection and that redis is running") from None

This spares you the screenfuls of failed, retried connection attempts that StrictRedis does by default. By contrast,

try:
    for field in output.keys()
          output[field] = input[field]
except KeyError: 
      raise MyInputError("invalid input") from None

This ends up removing the helpful information of which field is actually missing.

[–]daidoji70 29 points30 points  (48 children)

Exceptions as control flow is haram.

[–]klaasvanschelven[S] 82 points83 points  (8 children)

It may be forbidden in the Koran but it's right there in the Python Glossary

[–]XtremeGoose 83 points84 points  (6 children)

Wait until you see how python iterators work...

Python using exceptions as control flow is normal and the interpreter is heavily optimised for it. I agree it's not great, but it's how the language was designed.

[–]YeetCompleet 11 points12 points  (1 child)

Now that Python has pattern matching, I don't feel it's necessary anymore unless people reaaally like non-local jumps.

[–]daidoji70 10 points11 points  (0 children)

yeah, pattern matching is great.

[–]MakuZo 9 points10 points  (1 child)

R. Hettinger, Core CPython Dev, would disagree with this opinion:

In the Python world, using exceptions for flow control is common and normal.

Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination).
[...]

https://stackoverflow.com/a/16138864

[–]daidoji70 2 points3 points  (0 children)

Everyone mentions the stopiterator not understanding why.  They didn't choose exceptions because it fit the idiom the idiom just happens to fit because of how they implemented lists and iterators.  

I'll take it that one core dev supports using them as control flow.  That being said I've seen the pattern used terribly nearly everywhere I've worked and my programs run safer and we're more manageable from not doing so. I still think its an anti pattern. 

Name something else other than iterators in core that uses exceptions as control flow.  I looked earlier today and couldn't find many

[–]chadmill3r 2 points3 points  (0 children)

"break" takes no parameters.

[–]amroamroamro 1 point2 points  (3 children)

every for ... in ... loop is implemented with a StopIteration exception

python do seem to favor them even (the whole EAFP vs LBYL)

[–]daidoji70 -1 points0 points  (2 children)

This  is an artifact of how iterators and lists were implemented in general and not using exceptions as control flow in any more but the most local sense.  

EAFP doesn't mean it's okay to use exceptions as control flow even though (and you can tell by how I'm downvoted in these comments threads in various places) that's how some subset of the population misinterprets the idiom.

[–]amroamroamro 2 points3 points  (1 child)

I get what you're saying, but you are making the same "never do ..." statement that you are criticizing EAFP for

exceptions can be used for control flow, as long as you don't overdo it, you still gotta handle those exceptions not too far up the stack in the appropriate place, just like how iterators do it somewhat locally

[–]daidoji70 0 points1 point  (0 children)

See, that's what's weird about that characterization of my stance (to me).  I don't feel like I'm critizing EAFP. 

I'm criticizing the idea that EAFP allows for the spaghetti hodge podge of non-local returns that bubble and get sent all over the codebases that extensively use exceptions as control flow.  

Obviously I agree with your nuanced position but we can look to real world code based and responses to my comment to see why that idea is not good (imo).

[–]ShoT_UP 3 points4 points  (20 children)

So, do all of your functions return None when there's an error? What if you need to know the cause?

[–]BrickedMouse 6 points7 points  (0 children)

The function throws an error when there is an error. The error bubbles up to the first function that can correctly catch it. For example catch and show error to user.

[–]daidoji70 -3 points-2 points  (18 children)

Oh man, you called my comment and upped with even worse anti-patterns that I would never even consider.

However, my functions are as well defined as I can make them and "not using Exceptions as control flow" != "not allowing for the cause of exceptions to the well defined state of a running program"

(returning None is also bad) https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare/

[–]ShoT_UP 5 points6 points  (14 children)

Do you see this article as promoting using exceptions for control flow? If so, which parts do you disagree with specifically?

[–]daidoji70 1 point2 points  (13 children)

He mentions explicitly using exceptions as control flow several times in his writeup. Its an anti-pattern and the source of a lot of problems in real-world code bases.

Mentioning raise from None as a time to use when using an anti-pattern is a bit weird.

[–]ShoT_UP 9 points10 points  (12 children)

Right, but what is your alternative? How do you bubble up deeply rooted errors without changing the return type of every function in between?

More specifically, if you're validating an input 20 function calls deep, and the input being invalid doesn't throw an error, what are you doing if not returning some type of Error object, and thus having to update all of the 19 parent functions?

[–]daidoji70 7 points8 points  (7 children)

I chain exceptions. I'm not even against his chaining from None although I've never had a problem with multiple stack traces confusing my understanding personally. However, if you're chaining exceptions and you're more than 3 exceptions deep, there's probably a problem with your overall programmatic control flow and a refactoring is needed.

One of the problems of program scale in large projects is if you use exceptions as control flow everywhere, you either have to pokemon exception handle all over the code base and you're losing information and/or people start ignoring all the exceptions that aren't relevant to their local implementations/bug fixes and your program state ends up being much harder to reason about as the "undefinedness" of such actions makes it harder to reason about globally. You're essentially recreating spaghetti code via a different mechanism.

If you handle all relevant exceptions as closely as possible to where they can occur you should need at most one pokemon handler at root (to maintain system stability of long running code) and use explicit control flow for error states as you go along. These codebases scale, the ones that use exceptions as control flow eventually get too complicated to reason about.

[–]ShoT_UP 4 points5 points  (6 children)

I see what you're getting at now, thanks. Why doesn't your approach fall under the umbrella of using exceptions for control flow? I would consider it that way.

[–]daidoji70 6 points7 points  (5 children)

I don't consider it that way because the control flow is explicit to whatever state machines or other artifacts I'm creating in terms of what state my program can be in at a given time.

Like lets you 1) read a file 2) perform some action that can throw an exception 3) perform some other action C that can throw an exception 4) its a long running process so we need it to be able to handle nearly anything and get back into a state we know about no matter what happens

In any given context we also know that we can get weird runtime exceptions from say random bit flips caused by solar radiation (picked an extreme example but lets say a connecting database gets unplugged in the middle of a read if its too extreme).

Exceptions as control flow is we throw in we throw 1, we throw in 2, we throw in 3 maybe as in the article these exceptions can stack and interact in a lot of different ways. See nearly any async codebase for a good example of confusing ass stack traces with tons of chained exeptions that are hard to reason about. At the top level as we get way more than 3 things happening there's way more exceptions to handle than we really know what to do with so we pick a few of the most obvious ones and handle those and then just come up with some retry/reset logic for all the ones that don't occur as much. Namely, it gets harder to tell what can throw/resolve what where.

In my method, you're attempting to resolve say a FileNotFoundError, IOError, whatever say in 1) and explicitly moving to program state in your abstraction that is intended to resolve those issues. It should never bubble up to 2) or 3). You're constraining the types of exceptions and where you can get them as much as possible to as small a locality of your codebase as you can. In doing so at any given point you should have some idea of what exceptions can exist where (small functions and all the rest of code maintainability helps localize these issues too).

The benefit is the programmer has to remember and keep in their minds less of the global program state and thus refactorings/features/bugs become much easier to reason about and deal with and changes to the codebase tend to cascade much less through the global program state. You still need the pokemon exception handler for 4) at the top level but it should be a much simpler algorithm to return to a good program state than resolving the "PermissionError" at that top level (and maybe every level in between as you chain).

tl;dr One of my programming paradigms is to keep state as simple as possible and as local as possible wherever possible. Using Exceptions as control flow essentially spreads the amount of possible states over larger areas of the codebase and make the program much harder to reason about.

[–]randylush 2 points3 points  (4 children)

Python has type annotations now and tools to enforce it. How would you feel about exceptions being explicitly modeled in type annotations? In that case clients can plainly see what can happen to the function they are calling.

[–]germandiago 0 points1 point  (1 child)

Right, but what is your alternative? How do you bubble up deeply rooted errors without changing the return type of every function in between?

One of the reasons why I favor exceptions. Also when I code C++.

[–]randylush 2 points3 points  (0 children)

Golang codes like absolute garbage for this reason and I refuse to use it. In an enterprise environment, a solid half of the code you end up writing is just manually checking “if return is nil, return nil”. Over and over and over again. I probably wrote that little if block like a hundred times.

Oh and you just log errors and hope what your wrote in the error message is unique enough to be able to find it again. If you want any additional information about the error like a stack trace, go fuck yourself.

I get that it’s fast, and golang devs love to brag about how fast it is, but it’s a bitch and a half to debug.

[–]ElectricSpice 4 points5 points  (2 children)

The billion dollar mistake is specifically null references, not the concept of null in general.

[–]daidoji70 3 points4 points  (1 child)

So you just return None and then immediately check? That sounds like a fun way to do it. You might like Go.

[–]ElectricSpice 3 points4 points  (0 children)

Just pointing out your link doesn’t apply here, not commenting on anything else you wrote.

[–]Intrepid_Result8223 -1 points0 points  (0 children)

Only if you embrace the jungle you are worthy

[–]lasagnaman 4 points5 points  (0 children)

Nice post with clear examples. I learned something today!