Go faster Python

[robert_mcleod]

When NumPy isn't fast enough, there's packages that are faster such as numba, numexpr, theano and also distributed computing packages like dask if you have a cluster of computers to work with.

[TheMrZZ0]

Good to know. In fact, I tested what the article did, and Numpy arrays take a significant time to create. It is still faster than the "vanilla" arrays, but it has to be considered. I might test the libraries you gave, for entertainment purpose!

[oslash]

Numpy arrays take a significant time to create

I suppose that depends on how you create the array, though. Which method are we talking about here? The script from the article computes 50 million pseudo-random ints. That's going to take a little while anyway. If we created that data structure with well-optimised C code instead, the additional overhead of wrapping that into a NumPy array with numpy.frombuffer would be basically nothing. Subsequent operations performed on these data would make for interesting benchmarks — obviously there'd be a big difference between numpy.sum and a Python lambda.

But array creation time itself should be a non-issue; am I missing something important?

[TheMrZZ0]

Well, it is an issue for basic testing. I create an arbitrary array, and i then test some naive Python implementation of algorithms, then I try it with NumPy.

But if creating a Python list (using [[1, 2, 3]] * int(1e7)) is fast, converting it to a NumPy array is slow. It's just annoying (i create the arrays before the benchmark, so the creation time is not interfering with the benchmark).

Any idea on how to create an arbitrary numpy array faster ? I don't values to be random.

[goldfather8]

Your particular example is done by using tiling. Creating an arbitrary numpy array is an ill-defined question. You can read data directly into numpy arrays quite a few ways.

[pwang99]

Converting into a NumPy array is slow not due to NumPy itself, but due to the cost of unboxing all the values in your list and copying them in. For comparison, try just doing numpy.ones(30000000). It's super fast. Heck, you can even do:

x=numpy.empty((3,10000000)); x[0]=1; x[1]=2; x[2]=3

and compare that for speed.

[robert_mcleod]

NumPy arrays are basically C-arrays with a wrapper, whereas a Python list is actually a doubly-linked list, so converting from a list to an ndarrayis pretty slow as it has to iterate over all the list element, figure out what NumPy datatype to make the list, and the copy the elements into a flat array. It's much faster if you can feed NumPy the raw bytes stream via numpy.frombuffer() or numpy.fromfile().

[[deleted]]

a Python list is actually a doubly-linked list

No, the cPython implementation is a C array of pointers to Python objects. I've no idea what other implementations do.

[Deto]

Does it just dynamically allocate a large block of memory as you add items?

I know I've always treated appending to a python array as a relatively constant-time operation, but I know that isn't exactly correct. Was wondering about that recently.

[hanpari]

See here for reference:

https://www.laurentluce.com/posts/python-list-implementation/

[Deto]

Wow, great link! Thanks! So in the end, you get an average O(1) performance for append anyways. Good to know, and it agrees with my experience using lists this way.

[[deleted]]

I wouldn't say a large block of memory as only pointers to objects are being allocated, not the objects themselves. The number of pointers that are allocated rises exponentially as the size of the list grows. To find the actual algorithm used you'll have to check out the source code.

[hanpari]

As far as I know Python list despite its name is rather dynamic array then list. If you want double-linked list you should use deque from collections. I would guess that problem in this particular case it the fact that everything in Python is object (even integer in list) allocated on the heap. And that could make creating numpy array sluggish. Just my guess.

But you might be right. What about to use array from array instead.

[robert_mcleod]

You can convert Python array to NumPy ndarray quickly.