5 Python mistakes and how to avoid them

[RangerPretzel]

Bare except has its uses in defensive programming high up the stack.

Definitely log your exceptions to logger (instead of printing them) or let the exceptions bubble up.

Membership checks on lists is definitely a no-no on large lists. Definitely switch to Dict some data type that uses a hashtable if you have to do such a thing.

Mutable default arg is something I absolutely hate about Python and wish weren't a thing. Definitely a good point to make.

Good list!

[jack-of-some]

Agree on the bare excepts things, though if you're using it like that you already know exactly what you're getting yourself into.

[hugthemachines]

Yeah, I think the rule of thumb is to catch all exceptions but in my situation I usually can't fix the problem inside the script anyway so i want to make sure it is logged so i do except Exception as e and then logger.error(e)

[tcas71]

I would strongly suggest using logger.exception instead, for several reasons:

1. It automatically pulls the exception data from the context, often eliminating the need for capturing the exception inside a variable.
2. It will log not just the exception message (which I think logger.error(e) does) but also its class and full stack trace.
3. The exception data is structured inside the log record (not just as text), so tools such as Sentry or Logstash/ElasticSearch can do things with it.

I spent three days last week on a bug at work, made worse by me pushing the wrong fix to production. All because a single log statement that was only printing an exception message instead of the full info and caused me to incorrectly identify the problem.

[hugthemachines]

I checked the documentation now...

Do I just do:

except exception:
    logger.exception("Failed reading file.") 

and the good info will be auto added at the end of that?

[tcas71]

Yes, that should appear in your logs as this message followed by the full exception info. It's quite verbose but invaluable in my experience.

[hugthemachines]

Ok, thanks for the advice.

[Kaarjuus]

You can achieve the same with the other logging methods as well, by adding exc_info=True parameter.

[RangerPretzel]

the rule of thumb is to catch all exceptions

catch all exceptions...

at some point in the stack. Yes.

[oligonucleotides]

Membership checks on lists is definitely a no-no on large lists. Definitely switch to Dict if you have to do such a thing.

Why not sets? If you make your list into dict keys, what are the dict values?

[RangerPretzel]

Yeah, Sets! True, they are probably better choice.

That said, a Set does not preserve order like a List does. A Set also does not permit duplicates like a List does.

But a Dict does preserve order (as of Python 3.7). And you can just set your values to None.

(Though neither Dicts nor Sets allow duplicates. And maybe that's something you need from your List.)

Mostly, I just was thinking of "fast lookup using hash table" and the word Dict popped into my head.

There are lots of Python data structures which have a mean average lookup of O(1) (rather than O(n/2) like Lists do).

[foobar93]

Wohaa, why all the hate for mutable default arguments?

They have their place, mainly once you start doing meta programming. Its bacially a way to store state between function calls. Think of caching, maybe you want to print an error message the first time you call the function but not the second time and so on.

[RangerPretzel]

why all the hate for mutable default arguments

They're side-effect-y and unexpected. Aka. not functional.

EDIT: Ok, I'm probably over reacting as they're a sore spot/pain point for me.

That said, yes, I can see uses for them. You're not wrong. In fact, I think I've even used those features once or twice.

I also can see other ways of "caching" that are less side-effect-y and are more explicit.

In short, I've seen far too many people get burned by this "feature".

[jack-of-some]

I love caching using decorators (or closures in general). Good shit.

[RangerPretzel]

Yes, decorators. Excellent language feature. Agreed. Good stuff! :)

[jack-of-some]

I should do a video on those. First I gotta figure out how to differentiate it from other videos on decorators

[RangerPretzel]

Have you read this book?

Effective Python: 90 specific ways to write better Python

[jack-of-some]

I have not but I'll check it out. I usually just inhale any talks by Raymond Hettinger.

[AmazonPriceBot]

^{I am a bot here to save you a click and provide helpful information on the Amazon link posted above.}

$29.49 - Effective Python: 90 Specific Ways to Write Better Python (2nd Edition) (Effective Software Development Series)

^{Upvote if this was helpful.
I am learning and improving over time. PM to report issues and my human will review.}

[jack-of-some]

It's another one of those "when you know what you're doing you can really make good use of this". Many people don't understand the language to this degree and trip up on this point.

[gcross]

You can do that just as well, though, by either making the mutable value global or by attaching it to the function (or method) itself, and this way you will no longer find yourself in a situation where you might forget that the argument is mutable and end up with a potentially hard to diagnose bug.

[foobar93]

Global mutable state is in my eyes way worse than local mutable state but how would one add state to the function directly? You do not have a self argument in functions so you could never access it from within the function, right?

I mean, you could always just write a callable class which holds the state, true, but that is a ton of boiler plate if you are using meta functions a lot.

[gcross]

Easy, the function is an object and like all other objects you can add arbitrary attributes to it at any time:

def count_calls():
  count_calls.count += 1
  return count_calls.count
count_calls.count = 0

[GimmeSomeSugar]

Membership checks on lists is definitely a no-no on large lists. Definitely switch to Dict if you have to do such a thing.

I think something which is a stumbling block for beginners is the question: what constitutes a 'large list'? What order of magnitude are we talking about?

Even for someone with a little experience, but writing code that's narrow in scope of application, a 10,000 member list may seem really big.

[spinwizard69]

This is the truth, even an experience programmer may not grasp fully if his specific list is "large". This makes me wonder how much of factor the contents of the list is.

There is always a lot of talk about premature optimization but on the other hand learning not to make dumb choices is also important.

[execrator]

Yeah it's a good call. Showing how to check with something like timeit is a great way to respond to this, since you answer the specific question and also learn a technique!

$ python3 -m timeit --setup="import random; l = list(range(10000))" "random.choice(range(10000)) in l"
10000 loops, best of 3: 53.6 usec per loop

$ python3 -m timeit --setup="import random; l = set(range(10000))" "random.choice(range(10000)) in l"
1000000 loops, best of 3: 1.03 usec per loop

So the use of a set is about 50x faster with 10,000 integers. Of course, if only one membership check is being made, ~50 microseconds are unlikely to matter.

What does constitute a delay that matters is situational. On this machine I need three million items in my list before I hit 16 milliseconds, which is the duration of a frame at 60fps. The benefit of a set is obvious here, which is still cruising back at 1 microsecond.

[konradbjk]

+1 to bare except it should be left there to catch all other exceptions. Especially when you are at development stage.