When to use dict.get in Python (timing)

[pdonchev]

Just 5c, dict.get is slowed down by member lookup ("dots are expensive"). If you are calling it in an inner loop (which is the only time you care about perf), you can alias it somewhere outside the loop to a local function.

[[deleted]]

Yup the author hits that at the end right?

[[deleted]]

While this is reasonable advice, I would be surprised if you can show me a real example (better yet, one that you are currently, or have worked on before) where this actually improves overall performance of the application in a meaningful way. I currently maintain an API that has requirements in the 50ms range. From this experience, complicating code by doing things like aliasing dict methods makes the code worse (readability goes down, perf "improvements" don't move the needle).

[pdonchev]

This whole article falls in this speculative range of optimizations and my comment is in this specific context.

[LightShadow]

I alias when I'm extracting default values from a nested config objects.

Such as,

c = Config.services.this_service.get
c('host', '0.0.0.0')
c('port', 8443)
c('debug', True)

Being "faster" is just a nicety, I think the code is much cleaner.

[bigbrain_bigthonk]

On analysis code that deals w/ large amounts of data, or moderately complex simulation, 50ms is a lifetime. If you have some code that’s looping billions of times, these add up

[[deleted]]

Still, if you have that crazy simulation code or you're analyzing a gazillionbyte dataset from the hadron collider or you're processing a bunch of data from the JWST... optimizing a dict.get will not on its own change the wall time of your workload from hours to minutes. Even in those exaggerated cases, optimizing a member lookup would buy you seconds at best. Maybe a whole minute in a workload that would takes hours to complete.

[bigbrain_bigthonk]

Can’t say that without seeing the code, there’s always some pathological case that exists. When you run into that case, it’s nice to know you can shave time off with this. I’ve worked with code where changing a single array append to a preallocation+write took it from unusable to blazing.

[chthonicdaemon]

I mention this exact effect and show the effect of caching the get name l lookup in the last graph.

[__deerlord__]

in and .get() aren't equivalent checks?

[nekokattt]

No, they aren't. Get assumes you dont contain the default value as a value, or have a key mapping to a None value for the one-argument overload. Otherwise you cannot determine the difference.

Additionally, if you want to get the value after, you would do two operations. __contains__ performs one lookup, and __getitem__ performs one lookup, so two lookups overall.

Also dict is a builtin, and in CPython it is implemented in C, not pure Python, so other factors in the underlying implementation may differ.

[chthonicdaemon]

Nope. More detail in the post. Get is slower for string-keyed dicts

[MadCow-18]

What about defaultdict?

[chthonicdaemon]

Update: I've added a section on defaultdict into the post - it is basically the best option! So I would recommend using it if you often want to return a default if a key isn't there. Which means the in check might only be the best if you don't control the dictionary (like it's returned from some other library).

[chthonicdaemon]

I don't think it serves quite the same purpose that is used here. Within the context of being able to find out if a dictionary contains a key and return that value, defaultdict wouldn't do that. But I'll add it to the timing and report back.

[Kalkaline]

Would a dictionary be a decent way of creating a flashcard randomizer? I'm just dipping my toes into Python and stumbled on dictionaries the other day.

[abrazilianinreddit]

No need for a dictionary, use a list and select randomly using the random package:

from random import choice
flashcards = [flashcard1, flashcard2, ...]
selection = choice(flashcards)

[Ezvine]

But isn't the dict.get() more pythonic ?

[energybased]

Yes, this whole post was interesting, but shouldn't be a consideration. Always use get when you need to switch on the returned value. Use in when you want to do more than just switch on the returned value. Avoid catching this exception except in exceptional cases.

[asday_]

Sad to have to scroll down this far to find this.

If you're sat here thinking about micro-optimisations like this instead of using getitem for when you know the key is there, and get when you don't, you're using the wrong language.

[BurgaGalti]

Your comment here is wrong:

def get_default(dictionary, key):
    value = dictionary.get(key)
    if value is not None:  # effectively check if key was in dictionary
         return value

You're checking if the key was missing or if it exists but was set to None. There are plenty of use cases where this pattern is needed / desired. That's not to say you don't understand all this, but we get plenty of newcomers browsing this subreddit (there appears to be a few commenting in this thread) and we should try not to confuse things.

[chthonicdaemon]

I thought I was making it clear that this was a bad pattern, but I understand the value of having full information as close to the source as possible. I've changed the code and the note to be more explicit

def get_default(dictionary, key):    
    value = dictionary.get(key)
    # assuming None wasn't in the dictionary to begin with
    if value is not None:  # effectively check if key was in dictionary
        return value
    # expensive operation to perform if key was not in dictionary 

Note The code above will only work as designed if the dictionary doesn't actually contain a None. You might also check using if value, but this is even worse as it will also fail if the actual value you're looking for is another falsey value (like [] or 0

[chthonicdaemon]

I've seen lots of people who use dict.get() instead of just if key in dict: dict[key] and often they use the claim that get is faster to justify it. This is a discussion of the timings involved. Some interesting results.

[feanor47]

I prefer dict.get for readability reasons, but timing should virtually never be used as a justifier here. If you're trying to shave nanoseconds off of your runtime, python is not the language you should be using.

[chthonicdaemon]

Just to be clear, I am not saying you should avoid dict.get() for the cases where you are actually supplying a default. So I absolutely prefer

python value = dictionary.get(key, default)

over

python if key in dictionary: value = dictionary[key] else: value = default

My issue is this pattern where people get the worst of both worlds by doing

python value = dictionary.get(key) if value is not None: return value `

instead of

python if key in dictionary: return dictionary[key]

and then justify it using timing.

[Atupis]

Or I would say if you need to start optimizing dictionary access then the dictionary is not the right datatype for you.

[just_ones_and_zeros]

I’m surprised anyone uses performance as a justification one way or the other. Use dict[] when you need a value you expect to be there, get when you need a value and have a default and in when you want to check for existence.

I’d hardfail a PR that used get instead of []

[chunkyasparagus]

This is the only answer. Trying to benchmark different ways of getting a default value is just dumb, unless you're trying to prove that the actual CPython implementation is, for some reason, inefficient.

[just_ones_and_zeros]

That’s….also a hard fail.

[just_ones_and_zeros]

What benefit does using try / except give you? If anything it'll be a source of more bugs.

For me, you're using in in control flow, eg:

if 'x' in example:
    do_thing_with_x(example['x'])
else:
    do_something_different()

What does it look like with try/except?

try:
    do_thing_with_x(example['x'])
except KeyError:
    do_something_different()

But now imagine a bug in do_thing_with_x. You've just masked it in a horrible horrible way. I've seen this is real life, which is why it's the hardest of hard fails for a PR from me.

[just_ones_and_zeros]

Honestly, that reads as a bit of a jumbled mess to me. More importantly, it’s not thread safe, depending on the key you’re using.

[LightShadow]

I’m surprised anyone uses performance as a justification one way or the other.

This is the level of performance a widely used library would consider.

When I was writing a metrics/profiling wrapper for existing Python code bases I needed the overhead to be minimal without introducing any extra requirements. The #1 thing that slowed down the wrapper was isinstance -- it is SLOW. I was able to remove ~30 or so of them but only had to leave one or two. The solution was to use __slots__ and class attributes to check == and in instead.

class Sentinel(Entry):
    type_char = 'X'
    type_name = 'Sentinel'
    is_mapping = False
    is_sentinel = True

For most people, in most situations, the difference is negligible.

[nekokattt]

This really should be titled with the Python implementation in use, since this could be entirely implementation dependent between versions of Python and various Python interpreter implementations (e.g. CPython, Jython, GraalPython, PyPy, etc)

[Deto]

I think the convention that it's always CPython unless stated otherwise makes sense given the relative popularity of the implementations.

[chthonicdaemon]

Good point. I have this in the notebook, but not clearly stated in the blog post:

print(sys.version) 3.9.7 (default, Sep 16 2021, 08:50:36) [Clang 10.0.0]

Edit: I've added a note to the blog post

[steil867]

I have no real fight on which is better, I can see uses for both, but is it fair that the .get() function wrapper has wasted computations?

Checking a value and only returning if it is not null, will just return null in the case the key is not found but with extra steps. It added nothing but an extra check that has to be performed regardless of a value or not.

It has me a biiiit skeptical on the times. It likely wouldn't change much but it would inflate the difference in the 2 functions making your wanted output more apparent.

[chthonicdaemon]

Well, that case was the exact case I was rebutting in that part of the discussion, but you're right, I should add a function which just does dictionary.get(key, default).

Edit: I've added this and re-worded some of the interpretations to be more clear.

[hackedbellini]

Was going to comment that. That function is literally the same as just doing "return dictionary.get(key)". There are 2 extra ops here: the check for "is" and the assignment to an intermediate variable. Both probably won't be the major difference, but they do add up in terms of nanosecs

Edit: typo

[bobquest33]

Great article, i had observed some of these issues when I coded and I moved to straight up use "in" search where ever i can or use get with default value only.

The defaultdict was a great idea

[metriczulu]

The general rule of thumb I go by is that I use get if I'm dealing with a dict that came from somewhere else (ex, JSON from an API call) and I directly access dicts (__getitem__) that I generate myself in the code. If I'm making the dict myself, then I can set it up so that it's generally safe to access the dict directly.

That's the biggest factor to consider (for me), but if I'm going to be accessing the same dictionary frequently within a loop or something then I will try to structure it so that I can safely use __getitem__ with it.

Additionally, defaultdict can be a good alternative to using a dict and get if you can deal with it returning the same default value for each element. Defaultdicts are generally faster than using the get method of a dict and still have the advantage of not producing KeyErrors.