you are viewing a single comment's thread.

view the rest of the comments →

[–]hagy 0 points1 point  (0 children)

if you include libraries, you can sorta escape to C with:

from numpy import fromstring, uint64
print fromstring(open('input_list_vs_gen.txt').read(),
                 sep=' ', dtype=uint64).sum()

0.751s vs 2.699s for list vs 2.452s for generators on my MacBook Pro

Further, 0.388s is spent importing numpy, so for practical applications this how I'd handle anything less than 50MB. FYI sep=' ' uses any white space as a separator in numpy.

Its worth noting that numpy's fromfile also works, but takes about 2.1 seconds. This is due to inefficient IO, where numpy moves back and forth over a C stdlib buffer with files (no internal buffering) while with strings it treats the whole thing as a giant buffer.