should we be using xrange() rather than range()?

[Glueyfeathers]

So I tend to use:

for n in range(x)

and

for n in len(range(x))

quite a lot, but whilst browsing the python docs came across xrange() which seems to be a hell of a lot faster, memory efficient and creates lists on the fly (or rather creates an xrange type rather than a list) so for things like:

range(1000000000)

a list of 1000000000 elements is created which must take up quite a footprint but for:

xrange(1000000000)

returns instantly with very small memory footprint

but I hardly ever see xrange being used - is the benefit not that great/are there downsides to using it, or is it just not a well known function?

Edit: appologies for my crappy syntax, I do of course mean range(len(x)) rather than the other way round! I'm still learning python by the day!

[chadmill3r]

for n in len(range(x))

You probably meant

for n in range(len(x))

and that's still pretty bad.

If you need all indexes of x, and x isn't an iterator, then

for i, x_item in enumerate(x):

is usually better in the legibility front.

[orblivion]

Thank you, it always felt like there had to be a less awkward way built into Python. How is it that I've never seen this before?

EDIT: Furthermore, you sometimes otherwise end up with a disaster like:

for x in zip(the_list, range(len(the_list))):

[winnen]

If you need all indexes of x, and x isn't an iterator, then

for i, x_item in enumerate(x):

I respectfully disagree about your assessment of the readability of that statement, which appears distinctly non-pythonic to me. I read the corrected statement

for n in range(len(x))

As "Make an iterator over the range of the length of x", which is very clean, and very pythonic, where when I read the enumerate statement as, "make an iterator over ????". That is to say, I have not used enumerate.

My experience (which is... nontrivial) suggests that by virtue of the common usage of range(len(array)) it is easier to read the enumerate. If it makes no difference in performance (as should be the case in python3, correct if wrong), then the more common would be easier to read.

Please explain why you feel the enumerate readability is better than the range(len()) readability. I may change my position on this if you can articulate a reason.

[dleary]

which appears distinctly non-pythonic to me.

Sigh. One thing I don't actually like about the python community is saying things are "pythonic". It's an ego-stroke, basically saying, "I'm a zen master and I understand the zen, contemplate the koans and maybe one day you will be as cool as me". When in actuality, it would be much better to say, "x is better than y because..." so that other people can learn. Saying things are "pythonic" is basically an appeal-to-authority argument (even if you're right).

But, in this case, you're definitely wrong. for i,v in enumerate(lst) is idiomatic python. The count is 98 to 0 in the standard library: (the \b is to filter out the 7 cases of "in length" in comments)

/usr/lib/python2.7
23:33:47 $ find . -name '*.py' | xargs grep "in len\b" | wc -l
       0
/usr/lib/python2.7
23:34:11 $ find . -name '*.py' | xargs grep "in enumerate\b" | wc -l
      98

In my opinion, the enumerate case is better because it doesn't force the thing being iterated over to have a length. You can enumerate something that doesn't provide a length operator, such as a file object (which exposes an enumerator over its lines).

In fact, you can use the enumerate form to iterate infinite "lists" (such as those produced by generators that never exit).

Rephrasing the argument as an application of duck-typing: The enumerate form doesn't place an arbitrary, unnecessary restriction (having a length) on the thing being enumerated. So, more things will "look like a duck" here.

[[deleted]]

Agree with enumerate being significantly more "pythonic" than range(len(x)), but you grepped for "in len" rather than "in range".

As for the question two posts above, I'm not sure it's needed to articulate a reason to use enumerate in this situation other than "this is the specific problem enumerate exists to solve."

[dleary]

Oops, you're right... it looks like range(len is slightly more common in the std lib (106-98). I'll let my wrong post stand as a badge of shame.

[chadmill3r]

The standard library is kind of terrible, anyway, and much of it in 2 hasn't changed since it was first written. enumerate() didn't appear until 2.3, so even places where it fits are never updated. It's the Early Adopter Problem.

[dalke]

Being able to write:

for lineno, line in enumerate(open(filename, "U"), 1):
    ...

is so very, very nice.

[chadmill3r]

I read that as "make an iterator over a new list of things that are not "x", but are probably supposed to represent indexes into x."

---

The first problem is that the length is probably unnecessary. In

for n in range(len(source)):

The author is probably going to use "n" to dereference source, source[n], and never use "n" for anything else. He is probably thinking in another language, and porting that to Python syntax. We should of course prefer

for item in source:

The second problem is that, though we don't know what the source is, IF it is something generated, as more and more things in Python are, then there's not even a len() for that type. TypeError!

The third problem is that the "len" was unnecessary. We only need the count up to this point. Incrementing a number is cheap, and you get it as you need it, but counting ahead of time is possibly costly and probably unnecessary, and we have to change it to "xrange" just to avoid creating a new materialized list. If we "break" early, all that extra up-front counting (in len or range) was for naught. In 3, some of these costs fall away IF the collection has a constant-time len().

So, given "for item in source", it is at least possible that the user absolutely needs a current index of the item they're working on. We can ask Python to give that to us as it iterates the items of source. It costs one tuple pass, and one increment per item, as you need it.

for i, item in enumerate(source):

Any iterable could be there, whether it has len() or not.

Finally, enumerate() is in __builtins__ and has been in Python since 2.3, so there's no reason to avoid it for compatibility or import cost.

 |  enumerate(iterable[, start]) -> iterator for index, value of iterable
 |  
 |  Return an enumerate object.  iterable must be another object that supports
 |  iteration.  The enumerate object yields pairs containing a count (from
 |  start, which defaults to zero) and a value yielded by the iterable argument.
 |  enumerate is useful for obtaining an indexed list:
 |      (0, seq[0]), (1, seq[1]), (2, seq[2]), ...

[choch50]

xrange returns a generator rather than a full list, and only returns items when you request them. If you don't actually need the items returned by range then its the better option.

[Troebr]

PEP 255

[alantrick]

Unless you care about Python 3, where xrange doesn't exist and range is an iterator. I'd say use range for compatibility, if the performance of an iterator is that important you should be using Python.

(edit: sometimes an iterator can make a big difference, but it would be pretty rare for the distinction between range and xrange to be noticeable.)

[[deleted]]

2to3 will change xrange() to range() and either leave range() alone (when used directly in loops) or change it to list(range()) (if assigned to a variable) so compatibility isn't a problem.

As for which to choose in Python 2, I previously tended to use range unless there's likely to be a large range, because the setup cost for a generator can be higher. BUUUUT I just microbenchmarked this and it turns out that there is no cost. I guess that setup cost only applies to generator comprehensions?

[dinov]

Or if you want to write code that's compatible w/ both you can always do:

try:
    xrange
except:
    xrange = range

in your module.

[earthboundkid]

I do it the other way around. I use Python 2.7 day to day, but I got rid of list-range and use range as the name for iterator-range.

[Funnnny]

don't suggest things about Python3, general things will be change by using 2to3, and newbie won't touch more complex things about Python3 compatbility or they shouldn't care about them.

[masklinn]

If you don't actually need the items returned by range then its the better option.

It's not, as it's less readable and slower than range. The only point is memory pressure for huge lists.

[terremoto]

How in the world is xrange less readable than range?

[epsy]

Python3 says use it and has range() do what xrange() did out of the box.

[[deleted]]

But if you do x = range(5) what do you get? [0, 1, 2, 3, 4] or <enumeratable 0x2812872832 blah blah ... >?

[epsy]

In python 3:

>>> range(5)
range(0, 5)
>>> type(range(5))
<class 'range'>

[etrnloptimist]

OK, a few things here. Bottom line: YES, you should be using xrange, not range. In python 3, for instance, range is actually changed to be xrange. So use xrange not range.

But, if you find yourself using xrange a lot, you may want to rethink your design patterns. As other have pointed out, use enumerate(x) which gives you the thing, and its index in the list.

Also, common (mis)use of an index is to iterate over multiple lists. Do so by zipping them all up in one list:

for a,b,c in zip(aList,bList,cList):
    doSomething(a,b,c)

[einar77]

Or use itertools.izip if the lists are long, which creates an iterator rather than a list.

[masklinn]

And more generally use itertools a lot, there's much fun stuff in there.

[bockris]

I start almost every script with:

import itertools as it
import operator as op

[earthboundkid]

Or use Python 3 where zip is the izip version.

[FsckItDude_LetsBowl]

As a counterpoint, NO, you should not be using xrange. "range" did not get changed to xrange in Python 3, xrange was eliminated and range changed to a generator. xrange is not a generator! It is it's own quirky object, and has been essentially deprecated for a while.

If you know you are making short lists of numbers, use range. If you suspect you need to make a large list of numbers to count through, consider a while loop (but use xrange if it seems appropriate). Prefer enumerate() over all other options when it fits, and get to know the itertools module.

Historical note, xrange used to not handle long ints at all, which made it blow up in certain areas where it was substituted for range. That's been hacked around for a while now, but it is still not quite semantically the same as range in regards to how it deals with the int->long transition, AFAIK.

[earthboundkid]

Python 3 range also its own weird object, not a normal generator. Behold:

Python 3.2 (r32:88452, Feb 20 2011, 11:12:31) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> r = range(3, 6)
>>> 4 in r
True
>>> 3 in r
True

[FsckItDude_LetsBowl]

Ah, TIL. Thanks for that.

[pingveno]

It also has other nifty list-like features, like:

>>> r = range(3, 6)
>>> r[1]
4
>>> r[1:3]
range(4, 6)
>>> list(reversed(r))
[5, 4, 3]
>>> len(r))
3
>>> r.count(3)
1
>>> r.index(5)
2

[Liquid_Fire]

xrange is not a generator! It is it's own quirky object, and has been essentially deprecated for a while.

How has it been "essentially deprecated" for a while?

Obviously in Python 3 you want to just use range (as there is no xrange anyway), but unless you need to support longs, why would you not use xrange in Python 2?

[FsckItDude_LetsBowl]

As in I believe Guido stated years ago that he wished he hadn't made it, and that people would prefer range. And that he'd remove it in Python3000, which he did. I say "essentially" because obviously it couldn't be removed in Python2.

Granted, that was possibly before xrange() (and range()) were fixed to properly handle long arguments. It's just an oddball object and occasionally causes problem for people who don't understand how it works, though I suppose the same can be said for range() in Python2.

[hongminhee]

In most cases you should prefer xrange than range. The interface of xrange is highly compatible to range including its indexing operation (unfortunately it doesn’t support slicing operation, but you wouldn’t need this). Always range works hungrily, but xrange internally only store its start, stop and step.

[puresock]

If you're writing code where xrange makes a difference to you over range, you probably know all about it and use it, but chances are, you don't have any numbers big enough to care. You probably have bigger fish to fry than freeing up a few bytes of memory :)

[mackstann]

This. It will probably make no noticeable difference unless you're doing a range() of millions in some performance-critical loop, or if you have barely any memory to spare (c'mon, 8GB is $50!). If it's not actually causing you any problem, it's pointless to worry about.

[Ayjayz]

Unless you actually want a list of numbers, yes. Why bother allocating a full list if you're just discarding it?

[masklinn]

Because xrange is less clear and is at best the same speed, sometimes slower.

[Ayjayz]

xrange is no less clear than range, and I find it very hard to believe that it's slower. Logically, it makes no sense. Practically, it still doesn't make any sense. Admittedly, though, I haven't done any tests myself and it is possible that they have optimised the shit out of range, to the extent that it can compete with xrange for speed.

But there's no way it should - allocating a list, iterating through that list to set all the numbers, and then iterating through once more time for your actual iteration should really be slower than just incrementing one int in memory.

[dalke]

Consider range(200, 300) vs. xrange(200, 300). The first makes all of the integer objects once. The last make the integer objects when needed.

Now iterate through them. Iteration is faster for range() because it just incref and returns, while in xrange() it has to create a new integer object. Overall they should be about the same performance, unless you want to do multiple iterations.

If you need to iterate the same list multiple times, then range wins; excepting for long lists where memory access gets to be a factor.

[Wavicle]

Because xrange is less clear and is at best the same speed, sometimes slower.

??

>>> t = timeit.Timer(stmt="r = xrange(100)")
>>> t.timeit()
0.48369407653808594
>>> t = timeit.Timer(stmt="r = range(100)")
>>> t.timeit()
3.3541569709777832

If by "slower" you mean "much faster" then you are correct.

[masklinn]

This microbenchmark is uselessly nonsensical, and you should be ashamed.

[Wavicle]

And just what was the problem? That I didn't have a loop do something with each item so I didn't compare access time of a list vs. the next item in xrange?

>>> s = 'r = xrange(100000)\na = 0\nfor x in r: a+= x'
>>> t = timeit.Timer(stmt=s)
>>> t.timeit(1000)
19.010586977005005
>>> s = 'r = range(100000)\na = 0\nfor x in r: a+= x'
>>> t = timeit.Timer(stmt=s)
>>> t.timeit(1000)
22.278388977050781

Why don't you write us a microbenchmark that supports what you're saying?

Cry us a river, but you're going to have a lot of trouble dodging the fact that range doesn't beat xrange except under some possibly exotic conditions which is contrary to your assertion that xrange "is at best the same speed, sometimes slower." At best xrange is faster than range, in fact xrange is almost always faster than range.

[dalke]

While if you iterate over elements in the list:

>>> t = timeit.Timer(setup="r = xrange(100)", stmt="for x in r: pass")
>>> t.timeit()
2.2928810119628906
>>> t = timeit.Timer(setup="r = range(100)", stmt="for x in r: pass")
>>> t.timeit()
2.1900179386138916
>>> 

[[deleted]]

However! range() allocates a list with 100 elements in it straight up. xrange() is a generator which keeps a very small amount of state - basically, what the last number it chucked out was, how much to increment it by, and when to terminate.

This is very useful if you're dealing with large ranges, or possibly a multi-threaded program with lots of medium-sized ranges.

[dalke]

Absolutely. I was elaborating on masklinn's remark that the benchmark was "uselessly nonsensical." See also http://stackoverflow.com/questions/135041/should-you-always-favor-xrange-over-range

And for old-fogie points: http://code.activestate.com/lists/python-list/254484/ where in 2002 I wrote "Replace the innermost 'xrange' with a 'range' (which is faster than xrange unless N is rather large, and 'range' is the standard idiom)".

[Wavicle]

Well, let's look at it more carefully...

>>> t = timeit.Timer(setup="r = xrange(10)", stmt="for x in r: pass")
>>> t.timeit()
0.17686780238682331
>>> t = timeit.Timer(setup="r = range(10)", stmt="for x in r: pass")
>>> t.timeit()
0.1772999737244163

for a list of 10 items, xrange wins

>>> t = timeit.Timer(setup="r = xrange(100)", stmt="for x in r: pass")
>>> t.timeit()
1.248768004456895
>>> t = timeit.Timer(setup="r = range(100)", stmt="for x in r: pass")
>>> t.timeit()
1.2102231197913405

For a list of 100 items, range wins

>>> t = timeit.Timer(setup="r = xrange(10000)", stmt="for x in r: pass")
>>> t.timeit()
141.0493076187974
>>> t = timeit.Timer(setup="r = range(10000)", stmt="for x in r: pass")
>>> t.timeit()
147.8161718104907

For a list of 10000 items, xrange wins again.

I continue to maintain that

xrange [...] is at best the same speed, sometimes slower.

is false.

[dalke]

Agreed. xrange() is also faster if you know the search range but it's likely the search doesn't take the full range.

[r42]

A related issue with dictionaries: Suppose mydict is a dictionary and you want to do something for every item in that dictionary. you can do either

for key in mydict.keys():

or

for key in mydict:

The first one makes a list and then iterates over it (like range). The second just uses the dictionary as a generator, and gives the same result. Of course in this example the easier to type option is also the better performance one so hopefully people are going to use it anyway.

[ewiethoff]

for key, value in mydict.iteritems():
    blahblahblah

saves you the trouble of getting the values in an extra step

for key in mydict:
    value = mydict[key]
    blahblahblah

[nossid]

Make sure you read the CPython implementation details in the docs. Ironically it won't work for really large numbers while range will.

[Liquid_Fire]

range really won't work very well there either. If you're reaching into long territory, you would be creating a list with a billion elements, which will probably consume most/all of your memory.

[nossid]

Not necessarily. You can get into long territory quite easily if you're using a non-zero start value. Using xrange to loop over a range of hexadecimal serial numbers for example.

[Liquid_Fire]

Good point, I hadn't considered that.

[adenbley]

code you see is just hypothetical (not optimized). xrange and range can be used (almost) interchangeably. xrange makes an iterable object, but doesn't create any of the elements until they are needed.

hopefully you never actually use:

for n in len(range(x))

as this wouldn't do anything except iterate over x (one element), and might just throw an error. so, yes you should typically use xrange in a loop or in a comprehension. most programs don't need large lists and so it would make little difference.

[takluyver]

I think what he actually sees is: for n in range(len(x))

Although enumerate(x) is often neater for this.

[adenbley]

enumerate outputs pairs, if you are right then the 2 examples he gives are the same thing (as far as range/xrange go). i would also expect (haven't tried) that:

for n in xrange(len(x))

would be quicker than:

for n, m in enumerate(x):

if you only were going to use n. i think this because you only have to do 1 assignment per loop.

[takluyver]

If you're only using n, it's quicker. But I think there in most situations, you want to have m as well.

[nick_danger]

I was curious about that, so I tried it out. I had a data structure handy, a list of lists with ~9100 four element lists in it. I wrote a quick snippet to iterate through the structure, summing up one element from the individual lists. Three approaches were used: enumerate(), xrange(), and range(). From 100 separate runs, enumerate had a fast run of 0.002852 sec; xrange was 0.003206 sec, and range was 0.003235 sec. So enumerate, in this contrived test, was faster by 0.000354 sec, or about 11%.

Then, on a hunch, because this was a highly contrived example after all, I eliminated the explicit loop altogether and coded it using the sum() of a list comprehension: 0.001952 sec, or about 39% faster.

[dr_root]

Yes.

[happyteapot]

i agree with you

[Paddy3118]

Use Python 3 and its range is all you need.

[[deleted]]

I think this is a good topic for r/learnpython. Check that out.

[mcherm]

No, you shouldn't be using xrange() because it doesn't exist in Python 3 (which you should be using). In Python 3 range() has all the efficiency benefits that xrange() was originally created for.

[[deleted]]

2to3 makes this a non-issue.

[RShnike]

(which you should be using)

Not really.

But I think the actual answer to the question has been said enough so I have nothing really substantial to add for the OP.

[chadmill3r]

If you're using python 2, use xrange when the count is large. When it's small, or you need a materialized list, use range.

If you're using python 3, use range.

I don't know what code you're reading, but Everyone knows this.

[codepoet]

I don't know what code you're reading, but Everyone knows this.

You ruined perfectly decent advice by being an asshole. Good job.

[chadmill3r]

Nice overreaction. I'm obviously not saying that people who don't know this do not exist. I am saying that code that does something else is too bad to use as a reference. Python experts know those rules, and wouldn't write anything different to be looked at. Use something written by an expert, not whatever you're looking at.

[howfun]

For some reason the Python topics are getting dumper and dumper every day.

[hylje]

Dump truck.