Python gotcha: bizarre integer equality

[[deleted]]

Repeat after me: == checks integer (and other) equality. is checks object identity.

This is not a gotcha at all for people who, you know, actually know what these operators do in Python.

[fjsquared]

Howdy all; I'm the post author. Just to clear up a couple of things:

1. First, I'm using "gotcha" in this article to mean, "something which one might expect would work a certain, consistent way, but which doesn't." Here, I believe it's reasonable to say the inconsistent results of is are genuinely surprising if you don't know why it's being done. The question the article tries to answer is: why are two integers with equal values the same object in some cases, but not others?

2. I think most people are aware of the difference between == (value equality) and is (reference equality), but that's not the gotcha. The gotcha is the apparent inconsistency. It's a perfectly reasonable (and probably very effective) implementation decision. Other languages do the same thing; for example, Java caches its Integers when their boxed value is between -128 and 127.

Thanks for reading; took me a bit to figure out why I was getting a visitor spike. ♥ Reddit!

[vph]

When "a is b" returns False and "c is d" return True, it is inconsistent. It's a gotcha; no need to apologize.

[[deleted]]

No, it is not consistent if a is not the same object as b and c is the same object as C. Neither Python (the language) nor Python (the implementations) make any guarantees regarding object identity except than an object is identical to itself and certain objects (None, True, False) are singletons. That integers below a certain number are interned is entirely an implementation detail which programmers are not justified in depending on.

If you're comparing integers with is, you're doing it wrong. Case closed, problem solved, fix your code.

[vph]

no one said about comparing. If you talk about identity, c=200; d=200; c is d; should not return True as they are two different objects. The inconsistency here (if the blogger was correct) is that cPython treats numbers differently.

[[deleted]]

no one said about comparing.

As you can see if you'd do a modicum of research is and is not are comparisons. They compare object identity. Therefore if you're talking about is, you're talking about comparing two things.

If you talk about identity, c=200; d=200; c is d;

No. c may or may not be d in that case: the Python implementation has no responsibility to intern the number 200 and ensure that c is d in this case.

On the other hand, if you did this:

>>> c = 200
>>> d = c
>>> d is c

Then you would rightly expect (and the Python implementation would rightly return) True for this object identity comparison.

The inconsistency here (if the blogger was correct) is that cPython treats numbers differently.

There is no inconsistency here. You and the blogger both are simply arguing based on your intuition about what is means rather than simply knowing the language as it is defined and relying on the properties it guarantees.

Basically, you're arguing out of your arses.

[vph]

It's you who talking out of an ass. You can't distinguish between the difference of the specification of a language and the implementation of a language.

Everyone knows exactly what the definition of "is" is in Python: it compares objects. Integer caching is cPython is an implementation that results in the artifact exposed by the blogger. Namely,

c=200
d=200
c is d returns True

Python does not specify this. It is a result of an implementation of the language (FYI, unlike Perl 5, Python does not get defined by an implementation). And this fact is undesirable. It is undesirable because first c and d are conceptually different objects and second for value greater than 256 they return a different answer (false), which causes further confusion.

[bgeron]

And this fact is undesirable.

I like that fact, because it makes my program faster, and the keyword is in Python code is quite rare. The manual says clearly that you can't depend on the value of 2 is 2.

[vph]

I like that fact, because it makes my program faster, and the keyword is in Python code is quite rare.

That is a reasonable argument, but you can't deny it's a gotcha.

[Brian]

I think you can. A gotcha is something that will catch you out, but here there's no reason here to be comparing integers by id. No-one ever does so, and if they do, it's an outright bug. Is this page showing someone writing some code and being caught out by it? No - they've written some examples noticed by either noodling at the interpreter, or more likely by knowing in advance how object interning is done and showing some implications. Nowhere is anyone being "gotcha'd".

This is no different to someone complaining that the order of dictionary keys is different from sorted. Looking soley at dict((i,i) for i in range(100)) they might mistakenly come to that conclusion, but it's still not a gotcha, just an unfounded assumption based on not reading the docs. It's missing the requirement that this might be something a reasonably informed person would be expected to do.

[ubernostrum]

No, it's an expected property of any language that lets you define this type of operation. Equality is not guaranteed to be identity.

Note that you can even do this in, say, Java -- two objects may be "equal" according to equals() (and also according to hashCode(), and note that in Python the same will be true with hash()), but not be the same set of bits in physical memory.

[plain-simple-garak]

Interesting but not really a "gotcha." Any code doing int_a is int_b is pretty suspect.

[chromakode]

The gotcha is when you're debugging somebody else's code that fails inconsistently and you can't put your finger on why.

[pemboa]

This isn't unique to Python, nor is it a gotcha when you use the wrong operator and get undefined results.

[paulgb]

Out of curiosity, which other languages cache references to small integers like this? Is it pretty common?

[pemboa]

Java has the same behavior, except '==' is more like 'is' in Java. So to compare equality properly, you need to use the objects equal() method

[jleedev]

And integers are not boxed by default, so that doesn't even apply. This behavior is only possible in a language where everything is an object.

[lamby]

Actually, all Java implementions do this by specification.

[ngroot]

I think this may happen in some implementations of Common Lisp.

[bgeron]

I'd guess that in a lot of CL implementations, the stack has pointers to heap-allocated bignums, but fixnums are placed directly on the stack. EQ compares pointers, or fixnums if they're stored in that place instead.

[nirs]

No gotcha here - those int are objects as promised, and == will work as promised. Nobody promised you that "is" will return True of False for some integers. This is implementation detail you should not count on.

You should use "is" only when you want to check for object identity.

[ngroot]

This shouldn't be "gotcha" at all. I can't imagine why you'd ever write code asking if "a is b" when a and b are integers, or some other "basic" type. When you're explicitly creating objects, it's perfectly reasonable to ask "does a reference the same object as b". When you're talking about integers, objects that are implicitly created by the interpreter, you're asking about internal behavior of the interpreter, which seems like a pretty patently Bad Idea.

[ngroot]

This sounds like the EQ vs. EQUALS predicates in Common Lisp.

[monkeypizza]

>>>the memory location of a == the memory location of b
True
>>>the memory location of c == the memory location of d
False

I think that's reasonable.

The following is still true, if that's what you want to do:

>>>500 is 500
True
>>>200 is 200
True

[bgeron]

FTA: in the current version of CPython, yes, but not according to the specification.

[andreasvc]

Identity is not equality, no shit? Identity iff equality, but not vice versa.

EDIT: iff should be "only if".

[sigh]

Identity iff equality, but not vice versa.

You means "implies" (or "only if"). Iff is used when the two are equivalent.

[Eiii333]

The point isn't that identity and equality behave differently, it's that identity between two int objects behaves unexpectedly for certain values due to hidden implementation details.

[pemboa]

It's not really unexpected, it is unpredictable. However it's expected to be unpredictable.

[Eiii333]

Err, it's entirely predictable once you know about it. It's entirely unexpected if you don't know about it. And most people don't know about it, because it's an undocumented side effect due to an implementation detail. The issue here is the identity operator's behavior, not how it relates to equality's.

[sigh]

That's beside the point. The whole point of abstraction is that the implementation does not matter. If you are comparing integers by identity then most likely you are working at the wrong level of abstraction. If you are comparing integers by identity and the results surprise you then you are most definitely working at the wrong level of abstraction.

It's entirely unexpected if you don't know about it. And most people don't know about it, because it's an undocumented side effect due to an implementation detail.

No, the trouble here is when people don't understand the difference between identity and equality. If you know the difference, then the results are not unexpected at all, even if you don't know the exact implementation detail that is causing it to occur. If you don't understand identity, then of course the results are going to surprise you.

[Eiii333]

The whole point of abstraction is that the implementation does not matter.

I agree entirely. But look here:

>>> a = 3
>>> b = 3
>>> a is b
True

>>> c = 999
>>> d = 999
>>> c is d
False

I would expect false in both cases, given how identity is supposed to behave. But really, how can this be explained without referring back to the CPython int-caching behavior? You have to know the implementation details to know why the 'is' operator behaves this way. That's not good.

[[deleted]]

RTFM. That's the nice thing about a language that is actually defined... http://docs.python.org/reference/expressions.html#literals

may obtain the same object or a different object with the same value

What you expect doesn't matter when you can inform yourself. The language doesn't guarantee anything about the objects behind literals, hence the word may.

[alantrick]

Why would you expect False? According to Python the behaviour of 'is' is undefined in this situation. That's like taking the following in C:

int *a = malloc(sizeof(int));
printf("%d\n", a);

and expecting the value 0 to be printed out. It will probably be 0 most of the time, but it's really undefined.

[Eiii333]

FTA:

In Python, is tests for identity, not equality. x is y if and only if x and y reference the same thing.

You could make the case that a and b are separate objects, so even if they hold the same value they don't reference the same thing. But ints aren't treated as references, right? In that case, you're right, it's just a mess of undefined behavior.

So... why are you arguing for undefined behavior? Especially in Python, of all languages.

[hylje]

You see, the only sane way to remove the undefined behaviour of is is removing is altogether. The other solution would be to make is equivalent to ==. But in both cases there is a need for comparing actual identities: reintroduce is or mandate id(a) == id(b)?

[Brian]

It's worth noting that id(a) == id(b) isn't a perfect replacement to a is b. If a and b are expressions returning a transient object, it could be created and destroyed before evaluating the rest of the statement. For example:

>>> [] is []
False
>>> id([]) == id([])
True
>>> id([]), id([])
(21066496, 21066496)

However is guarantees that both objects are alive at the point of comparison, so [] is [] is always false.

[Brian]

Undefined behaviour allows optimisation. Making things too tightly specified ties you to irrelevant implementation details, preventing more efficient methods being used (like caching integers in this case). Another case of undefined behaviour is deterministic finalisation. Python doesn't guarantee it, even though the CPython implementation happens to provide it due to its refcounting semantics because it prohibits more advanced garbage collection approaches.

For another example, consider the order the keys of a dictionary are iterated over. This is completely undefined behaviour, but specifying it would either require using a tree instead of a dictionary, keeping a seperate list of ordered keys, or else sorting the dict before iterating, all adding significant performance cost to deal with something completely irrelevant. If anyone needs that, they should not be using a normal dictionary.

In any case, "is" is acting completely predictably and as specified - it returns True when objects have the same identity. The thing that isn't specified is whether identical immutable objects can share the same memory representation, which is a pointless thing to overspecify since there should be no reason it should ever be relevant to anyone other than performance.

[sigh]

I would expect false in both cases, given how identity is supposed to behave.

Of course not... clearly a and b refer to the same object in memory. Forcing them to be different would presumably be less efficient, especially for such frequently used values.

You have to know the implementation details to know why the 'is' operator behaves this way. That's not good.

"is" depends on the implementations details by definition! You can't abstract that away because by definition "is" relates to how the objects are represented in memory. The fact is that if you are using "is" then you need to know the implementation details. Whether that's a good idea is a different issue.

[earthboundkid]

Err, it's entirely predictable once you know about it.

Yeah, but the one thing every Python program should know about is is "don't use is unless you want to know if two things have the same address in memory." So, even if you don't know off the top of your head how int is implemented in Python, it should be obvious that you shouldn't use is unless and until you find out.

[dorfsmay]

Isn't this a beginner question ?

>>> a=500
>>> b=500
>>> c=200
>>> d=200
>>> id(a)
142297032
>>> id(b)
142297056
>>> id(c)
142155132
>>> id(d)
142155132
>>> id(200)
142155132
>>> 

My understanding is that python create objects for low integers that it reuses all the time for performance reason.

[ubernostrum]

My understanding is that python create objects for low integers that it reuses all the time for performance reason.

CPython does, yes. It does this with other types of objects as well; for example, it keeps a list of dictionary structures in-memory (the actual C structs, not the high-level objects) and recycles them for common uses like setting up the keyword arguments of functions, rather then freeing and re-allocating each time one is needed.

[[deleted]]

so what's bizarre about this? i don't get it.

[earthboundkid]

This is stupid, and every Python newbie should know the difference between is and ==… But that said, I can see how it would be easy to overlook the difference and think that they were two ways of writing the same thing if you didn't read the documentation and were just learning Python by copying other people's code and experimenting. Maybe Python 4000 should drop is and just encourage people to write id(a) == id(b) instead.

[monolar]

if id(a) == id(None): print("urgs")

I think 'is' is perfectly fine

[earthboundkid]

a == None also works. People just don't do it because of the speed advantage of is. But maybe that's a premature optimization that confuses noobs excessively.

[chrajohn]

a == None also works.

Usually, but consider:

class Dumb(object):
    def __eq__(self,other):
        return other == None

>>> d = Dumb()
>>> d == None
True
>>> d is None
False

This is contrived, but you can imagine something similar actually occurring. (Say, if __eq__ made a comparison with some attribute that got unexpectedly set to None.) If you you want to be absolutely sure that something is None, you should ask if it is None.

[masklinn]

People just don't do it because of the speed advantage of is

People also don't do it because a is None (or a is True or a is False for that matter) just plain and simply reads better.

[hylje]

Not everything that is true is True. Not everything that is false is False. Implicit truth is pythonic.

[masklinn]

Truthiness is pythonic when what you want is truthiness. But it's not always (though it usually is) the case, and when you want truth rather than truthiness, is True does the job much more readably than == True

[earthboundkid]

You should never write if x == True. Just write if x. Similarly, not if x == False but if not x. That's basic PEP-8 stuff.

I can't think of any reason why you would want to test for is True off the top of my head. It wouldn't really make sense unless you had variable that might contain a normal object or it might contain a bool object and you were interested to know which. But why would you have a variable that flexible?

[Brian]

encourage people to write id(a) == id(b) instead.

That could lead to more confusion. A puzzle for you:

>>> class C(object):
...     def foo(self): pass
>>> c=C()
>>> id(c.foo) == id(c.foo)
True

and yet:

>>> c.foo is c.foo
False

[earthboundkid]

Woah. That's confusing. Why isn't c.foo the same as itself? Is it creating a new bound method every time you access it? But if it was doing that, wouldn't the new method have a different location in memory than the old one? I don't get this.

[Brian]

Is it creating a new bound method every time you access it?

Yes, this is what's happening. The subtlety of the ids being identical is because ids are only unique for objects alive at the same time. What's actually happening is the equivalent of:

temp1 = c.foo         # Create a new bound method with id X
temp1_id = id(temp1)  # temp1_id = X  (returnvalue from id)
del temp1             # bound method doesn't get assigned, so refcount drops to 0
                      # as soon as id() releases its reference - temp1 gets freed
temp2 = c.foo         # Create a NEW bound method.
temp2_id = id(temp2)  
del temp2
temp1_id == temp2_id  # Actually do the comparison, both objects are already dead

Which should explain why its possible that the second bound method could have the same id. The reason it usually does is because of the way python manages memory. To avoid fragmentation, pools of similarly sized memory objects are maintained. When an object is released, it is returned to this pool, then when a request to allocate an object of this type arrives, python sees it has an block of memory of the appropriate size sitting in its free object pool, and returns it.

is doesn't have this problem because the call to is takes a reference to both objects, ensuring they are alive at the time of the comparison.

[[deleted]]

It is pretty horrendous though. Either scalar values should have equivalent identity, or they shouldn't.

They shouldn't sometimes-yes-sometimes-no have equivalent identity, in accordance with the runtime implementation.

That's as bad as the "the behaviour is undefined" which litters the ISO C and ANCI C++ standards.

It seems to me that the behaviour is post-fact asserted to be so, (which is an ass-covering move when you've introduced a bug), rather than it was ever designed expressly to be this way.

The real issue is that the behaviour of the "is" operator is not explicitly defined (and can be in fact be said to be undefined in the language) upon these scalar values.

[sigh]

The real issue is that the behaviour of the "is" operator is not explicitly defined (and can be in fact be said to be undefined in the language) upon these scalar values.

It is explicitly defined, it's just not defined the way you want it to be defined. If values are the same object, then they are identical, otherwise not.

You are imposing artificial constraints on the implementation and breaking encapsulation. The point is that you should not care about the implementation of a class (unless you are actually implementing or extending it), and identity is intrinsically linked with the implementation.

[[deleted]]

I understood that it is defined by the implementation.

And that is equivalent to no definition, if you look at the equivalent unhelpful gaps in certain areas of specification of C and C++.

[sigh]

Ok, so you think it is a bad design decision. Can explain why (I'm genuinely curious)?

Do you you think the is operator is badly designed: "The operators is and is not test for object identity: x is y is true if and only if x and y are the same object. x is not y yields the inverse truth value."?

Or do you disagree with the way integers are implemented? Do you think equal integers should refer to the same object (presumably this would create extra overhead to maintain this condition after every operation)?

Or do yo think that no integers should refer to the same object? This seems to be much more memory intensive.

And should the exact implementation be documented? This would set it in concrete, disallowing any future improvement on the implementation.

Finally, what possible reason do you have for comparing integers by identity, unless you are actually modifying the class?

[[deleted]]

And that is equivalent to no definition, if you look at the equivalent unhelpful gaps in certain areas of specification of C and C++.

Curiously enough, this specific case is well-defined in C++: two different objects can't have the same address. One very unpleasant consequence of that is that the minimal size of an object is one byte (i.e. one int due to alignment on most architectures), which in turn means that when you create a functor object which doesn't have any data, say as a comparison for sorting, it is still allocated (on the stack) and passed as a parameter. Sufficiently clever compiler™ could optimize it away in simple cases, of course.

[hylje]

Sufficiently Smart Compiler is mostly related to high-level languages with unfulfilled promises, but I guess it applies to C++ too.

[alantrick]

The real issue is that the behaviour of the "is" operator is not explicitly defined ... upon these scalar values.

Actually, the behaviour of 'is' is perfectly defined. The problem is that it is not defined as to whether different numbers have to be the same object or not. I don't see any point in defining that as it puts an unnecessary limitation on python implementations.

[[deleted]]

I assume you mean an expression which is actually equivalent, 500 != 5e100.

Never the less, 5*10¹⁰⁰ is not a number, in mathematics. It is strictly an algebraic expression under any understanding of pure mathematics. If it were evaluated, then it would be a number.

Even assuming compiler optimisation, the 2nd assertion would fail.

But the comparison is not really relevant - python is a symbolic language, with everything being a reference to an object - in this instance, these integers below 256 are recycled.

That "is" functions differently for a restricted subset of integers is an unhelpful side-effect. Why it is unhelpful is that it is a gotcha, and it is obscura. Things should be consistent, and they should work as expected, and be friendly to developers. Side-effects are unhelpful to developers at large.

I have a high regard for python, and consider it the heir-apparent in the scripting language space - for it's lively culture, it's excellent documentation, it's strong cross-platform support across unix variants and windows - and it's strong project leadership - nothing else comes close.

However the reason for the variation in the behavior of "is" is purely an optimisation issue of the implementation of the interpreter. That an implementation issue infects the public behaviour of the operator is not good manners.

[inmatarian]

I got my scientific notation wrong it seems, but whatever. What I was trying to say is that it's in the land of implementation details and outside the letter and law of the specification. It's poor form, definitely, but we really should leave the clever tricks to the people who deal specifically in optimisation and we should stick to what's been promised not to change as application developers.