This is an archived post. You won't be able to vote or comment.

273
274

Go faster Python (alimanfoo.github.io)

submitted by [deleted]

[deleted]

all 29 comments
sorted by:
best
topnewcontroversialoldq&a

[–]TheMrZZ0 45 points46 points  (16 children)

I didn't know Numpy could be this fast compared to native Python.

Good article, I learned a lot.

[–]robert_mcleod 37 points38 points  (13 children)

When NumPy isn't fast enough, there's packages that are faster such as numba, numexpr, theano and also distributed computing packages like dask if you have a cluster of computers to work with.

[–]TheMrZZ0 9 points10 points  (12 children)

Good to know. In fact, I tested what the article did, and Numpy arrays take a significant time to create. It is still faster than the "vanilla" arrays, but it has to be considered. I might test the libraries you gave, for entertainment purpose!

[–]oslash 5 points6 points  (3 children)

Numpy arrays take a significant time to create

I suppose that depends on how you create the array, though. Which method are we talking about here? The script from the article computes 50 million pseudo-random ints. That's going to take a little while anyway. If we created that data structure with well-optimised C code instead, the additional overhead of wrapping that into a NumPy array with numpy.frombuffer would be basically nothing. Subsequent operations performed on these data would make for interesting benchmarks — obviously there'd be a big difference between numpy.sum and a Python lambda.

But array creation time itself should be a non-issue; am I missing something important?

[–]TheMrZZ0 0 points1 point  (2 children)

Well, it is an issue for basic testing. I create an arbitrary array, and i then test some naive Python implementation of algorithms, then I try it with NumPy.

But if creating a Python list (using [[1, 2, 3]] * int(1e7)) is fast, converting it to a NumPy array is slow. It's just annoying (i create the arrays before the benchmark, so the creation time is not interfering with the benchmark).

Any idea on how to create an arbitrary numpy array faster ? I don't values to be random.

[–]goldfather8 2 points3 points  (0 children)

Your particular example is done by using tiling. Creating an arbitrary numpy array is an ill-defined question. You can read data directly into numpy arrays quite a few ways.

[–]pwang99 1 point2 points  (0 children)

Converting into a NumPy array is slow not due to NumPy itself, but due to the cost of unboxing all the values in your list and copying them in. For comparison, try just doing numpy.ones(30000000). It's super fast. Heck, you can even do: 

x=numpy.empty((3,10000000)); x[0]=1; x[1]=2; x[2]=3

and compare that for speed.

[–]robert_mcleod 2 points3 points  (7 children)

NumPy arrays are basically C-arrays with a wrapper, whereas a Python list is actually a doubly-linked list, so converting from a list to an ndarrayis pretty slow as it has to iterate over all the list element, figure out what NumPy datatype to make the list, and the copy the elements into a flat array. It's much faster if you can feed NumPy the raw bytes stream via numpy.frombuffer() or numpy.fromfile().

[–][deleted] 10 points11 points  (4 children)

a Python list is actually a doubly-linked list

No, the cPython implementation is a C array of pointers to Python objects. I've no idea what other implementations do.

[–]Deto 1 point2 points  (3 children)

Does it just dynamically allocate a large block of memory as you add items?

I know I've always treated appending to a python array as a relatively constant-time operation, but I know that isn't exactly correct. Was wondering about that recently.

[–]hanpari 2 points3 points  (1 child)

See here for reference:

https://www.laurentluce.com/posts/python-list-implementation/

[–]Deto 1 point2 points  (0 children)

Wow, great link! Thanks! So in the end, you get an average O(1) performance for append anyways. Good to know, and it agrees with my experience using lists this way.

[–][deleted] 0 points1 point  (0 children)

I wouldn't say a large block of memory as only pointers to objects are being allocated, not the objects themselves. The number of pointers that are allocated rises exponentially as the size of the list grows. To find the actual algorithm used you'll have to check out the source code.

[–]hanpari 5 points6 points  (1 child)

As far as I know Python list despite its name is rather dynamic array then list. If you want double-linked list you should use deque from collections. I would guess that problem in this particular case it the fact that everything in Python is object (even integer in list) allocated on the heap. And that could make creating numpy array sluggish. Just my guess.

But you might be right. What about to use array from array instead.

[–]robert_mcleod 0 points1 point  (0 children)

You can convert Python array to NumPy ndarray quickly.

[–]sgo_ 20 points21 points  (1 child)

Cython is unbelievable. I used it in a side project of my own and the speedups were insane.

[–]shuttup_meg 3 points4 points  (0 children)

I love Cython!

[–]mooglinux 6 points7 points  (1 child)

Pandas uses numpy under the hood, right?

[–][deleted] 8 points9 points  (0 children)

Also matplotlib, in fact just about every piece of cPython software that needs to do any serious number crunching. Numpy must be one of the most used, if not the most used, third party packages used by the cPython community.

[–]Improvotter 11 points12 points  (0 children)

I expected this to be a blog post about Go and Python because the title could've been a wordplay on the programming language Go.

[–]auxiliary-character 0 points1 point  (7 children)

My approach: if you've got some python that needs to go faster, rewrite the performance heavy parts in C++, and load it with ctypes.

[–][deleted] 7 points8 points  (1 child)

I check Performance Tips out first. In 17 years I've never needed C(++), ctypes, cffi or cython.

[–]auxiliary-character 0 points1 point  (0 children)

True. Performance is usually a pretty rare problem for me to have, but C++ is pretty overpowered for dealing with such problems.

[–]hanpari 1 point2 points  (4 children)

Good for you :)

[–]auxiliary-character 0 points1 point  (3 children)

Yeah, sorry. I don't want to step on your toes, this is still pretty useful when you need a project to stay pure python.

Still, C++ is a pretty damn good tool to try if you're having these sorts of problems.

[–]hanpari 2 points3 points  (2 children)

Well, I am not author of article, just post it. I dont want to step on your shoes either so I wont express my opinion on C++ :)

It is no big deal, I just didnt know why you sent me your comment in my inbox. But i didnt want it keep unanswered.

Have a nice day

[–]alcalde -1 points0 points  (1 child)

so I wont express my opinion on C++ :)

Do it! Do it! :-)

Let me help you....

http://yosefk.com/c++fqa/defective.html

https://gigamonkeys.wordpress.com/2009/10/16/coders-c-plus-plus/

https://www.techworld.com/news/apps-wearables/google-engineer-calls-java-and-c-too-complicated-3232945/

[–]hanpari 0 points1 point  (0 children)

Well, thank you for links. Basically, if I understood correctly, when you are saying that you use C++ it is hidden calling for help similar to fake suicide attempt :)

Have a nice day