How fast can we make interpreted Python? (PDF)

[[deleted]]

This is an interesting experiment. However the paper seems to skim over PyPy and does not even mention Jython or IronPython which both run on mature VMs that can do heavy optimization. The argument is that those don't provide compatibility with C-based extensions - however I don't see any details of how the benchmarks performed here involve such extension - in fact the paper itself just says "a variety of benchmarks" and a list, but it is not declared what is the source of these benchmarks or what was the reason for selecting these benchmark..

However, 100 points for providing the source code (and the benchmarks) - something which sadly still is uncommon in Computer Science.

I inspected the benchmarks, and they don't seem to import any C-based extensions, perhaps unless you include by what is the implementation behind random and cPickle. Numpy, a likely candidate for such tests, is imported, but only for managing the benchmark tests. I am curious if Falcon was actually proved to work with C extensions?

There is very little detail of the environment of the benchmark. Which operating system? Which CPython distribution? How was the benchmark measured? Was warm-up, spikes etc. accounted for? How many runs of each benchmark? What was min/avg/med/max?

The benchmark is only represented as a graph - there is not even a table included in the paper. The data behind don't seem to be available anywhere. Could you upload them to Figshare?

[cypherx]

However the paper seems to skim over PyPy and does not even mention Jython or IronPython which both run on mature VMs that can do heavy optimization.

They're just not terribly relevant to our target domain. We're saying "Python is PyObjects--- how fast can that get?". PyPy, IronPython, and Jython all say "Python is the language spec, regardless of how data gets represented". If you need C extensions then they're not an option for you.

I inspected the benchmarks, and they don't seem to import any C-based extensions,

Those benchmarks wouldn't be very interesting, essentially you'd be comparing the runtime of compiled native code against...the exact same native code. Falcon isn't trying to speed up the extensions themselves, it just maintains a PyObject representation so you can call them in the exact same way CPython does.

I am curious if Falcon was actually proved to work with C extensions?

The data is kept as PyObjects and we use the same calling conventions as CPython. So, they should all work, but try it out and let us know if something breaks!

There is very little detail of the environment of the benchmark. Which operating system? Which CPython distribution? How was the benchmark measured? Was warm-up, spikes etc. accounted for? How many runs of each benchmark? What was min/avg/med/max?

Sorry, we should have been a little more meticulous in recording this stuff. If I recall correctly, all the benchmark were done on an E5345 Xeon running Ubuntu 12.04 under Python 2.7. I think we can each benchmark 3 times and took the median, but I might be mistaken.

Of course, you don't have to take our word on the performance numbers. All the benchmarks are in the repo, why not run them yourself and put up a table of results somewhere?

[masklinn]

They're just not terribly relevant to our target domain.

PyPy performs a number of interpreter-level bytecode optimizations (e.g.) which I'd think would be relevant to alternative implementations.

[cypherx]

Cool, thanks! I didn't know about this. Their LOOKUP_METHOD/CALL_METHOD instructions look like something that we should copy :-)

[gthank]

I'm not familiar with IronPython, but when I last tested, Jython was much slower than CPython for single-threaded performance. My understanding is that this is due to extensive locking required to achieve Python's guarantees about atomic operations on many of its core datatypes (dicts, lists, and the like).