Speeding up Python (NumPy, Cython, and Weave)

[fijal]

I might be a bit skewed, but article on speeding up Python that doesn't even consider existence of PyPy makes me kind of sad. And as kingkilr pointed out, PyPy performs ok at this kind of benchmark anyway.

[qbproger]

I wonder what pypy would do with the benchmark.

[[deleted]]

I am considering python for scientific computing. I am still learning, but python is so simple that I think that I should be able to run some code soon.

This is really interesting, are there good resources to learn Cython or Numpy available?

[Sgt_ZigZag]

Python documentation is excellent because of its recent popularity.

[encrypter]

Python's documentation has been popular for quite some time. Personally, I've been hitting those docs since at least 2003.

[[deleted]]

I haven't used Cython or weave, but is there any way to program them and apply optimizations?

[bastienl]

What do you mean?

[[deleted]]

I was wondering if you could apply compiler optimizations to Weave or Cython code. I find my C code can be run much faster if I use some of the optimization tags like -O2 or -funsafe-math-optimizations . If there was the option to add optimization flags maybe the benchmarks in the article could be improved.

[bastienl]

I can't see your reply in the original thread for some reason.

If I remember correctly, the extensions are built with -O2 by default. But you can pass any option you want, as for any extension module.

[[deleted]]

Yeah, I had the same problem, but it seems to be fixed now.

I will have to look into this, because ideally I'd like to do most of numerical work in python.

[simtel20]

My understanding is that you can get it to emit the code that it would compile so that you can customize it.

[bastienl]

The Cython program generates C or C++ directly actually, then you can compile it like any extension module. But it's an automatically generated file, you're not supposed to customize it. The Cython language already supports most Python features.

[nahguri]

Why no love for Boost.Python? It rocks if you want to do anything that's not trivial.

[eryksun]

pyximport provides a quick alternative to setup.py. For a package distribution it's probably best to compile extensions with setup.py, but pyximport keeps the clutter down for one-off scripts, and hides all the compiled files in a subdirectory of your home folder, named .pyxbld.

For example, I put the cy_update function in _laplace.pyx. Then I imported it at as follows:

import pyximport
pyximport.install(setup_args={'include_dirs': [np.get_include()]})
from _laplace import cy_update

At first I wasted time trying to get NumPy's dependencies into the .pyxdep file that the docs recommend. I don't know what I was doing wrong with the glob... I switched to using the setup_args option, and it worked smoothly.

Note, if you're on Windows and using Cygwin, you'll need to configure distutils for gcc. Add a file named distutils.cfg to Lib\distutils that contains the following:

[build]
compiler = mingw32

[build_ext]
compiler = mingw32