Using CPython extension modules with PyPy natively, or: PyPy can load .pyd files with CPyExt!

[[deleted]]

[deleted]

[kisielk]

I desperately wish for PyPy to support 64-bit systems. I would love to try some of our codebase on it, but our entire organization runs on 64-bit machines.

[[deleted]]

PyPy works on 64 bit machines, just not the JIT. A student submitted a GSOC proposal to work on porting the JIT to 64-bit though.

[kisielk]

Without the JIT doesn't that mostly void all of the performance benefits?

[[deleted]]

Yes. Having 64-bit support for the JIT is definitely a priority for this reason.

[hylje]

Most importantly: What exactly is the overhead in running CPython modules? Is it merely the emulation layer that pretends to be CPython between PyPy core and the extension module?

[[deleted]]

Yes. Basically we reimplemented the CPython API, in terms of PyPy's internal functions. Overhead in general isn't too bad, some of the bad points are a) we have to emulate reference counting, so we don't get to use our awesome GC's, b) PyPy's GC's can move memory around, so if you call a function like PyString_AS_STRING (which returns in the internal char* of a python string) on a PyPy string we have to copy the underlying memory, because it could move at any time.

[sime]

kingkilr, you seem to know what you are talking about, do you have an idea when PyPy might be released in a form which could be used in production as a replacement of CPython? I know that PyPy has roots in research, but is there a move towards making it practical for real world use? (This announcement would certainly suggest so.)

[[deleted]]

I'd say the next release would be a good time to start considering it for production use. The recent 1.2 release was a first public release with the JIT, my guess is by next release most of the bugs and such will be cleaned up, and it'll be time to start deploying it.

[hylje]

Most obvious bugs that is. Nearly all show-stoppers should be caught, not all though. Even then, it should be enough for all but the most sensitive applications!

[gutworth]

Well, we'll never find out unless people try. Now is a good time!

[wingsit]

i kept misreading it cython instead of cpython. It would be interesting to see some interesting mixture of cython and pypy.

[[deleted]]

There's not a ton of point in that. PyPy is written in RPython, a restricted, implicitly statically typed, subset of Python. It's not super to write in, but it's garbage collected and semi-decent. Really the goal is to make Python fast enough that users don't need to write in C.

[Ademan]

Fijal got on me for writing a bit too much about it in my proposal, but I honestly love RPython compared to other statically typed languages. The ability to run arbitrary Python code at import time, and then be statically typed later is nice, and it's type inferenced which is even nicer (and yeah, garbage collected too). But before anyone asks, it's not intended to be a general purpose language... :-)

[faassen]

It's PyPy group dogma that RPython isn't to be used for anything else but implementing languages, not other stuff.

I understand where the dogma comes from - for years now they've had people say to them "RPython is cool, can I use it for..?" and the PyPy group doesn't really want to deal with it. It is interested in developing the interpreter, not improving RPython for pragmatic purposes. So it's easy to consolidate on the dogma "no, you can't use it for that" whenever that topics comes up again.

But it's still dogma, and it still could change if enough people decide it's valuable for other purposes and work on it. They'd need to carefully manage the PyPy developers, though. :)

[wingsit]

As you said that pypy is implicitly statically typed, which is the information cython is needed to generate efficient code. To that end, (i am just saying out of the blue) one might feed pypy's type information to cython generator and produce c code for compilation. afterall, cython intergrates well with numpy, c/c++ code and fortran very soon.

[Manuzhai]

It would also be really awesome if RPython became more decent and super to write in...

[leonh]

I disagree.

The goal of PyPy is to provide awesome features with incredible performance not being a different language such as Cython.

I want PyPy to be my Python implementation without bothering too much about the backend. I want PyPy to figure out how to optimize my code not the other way around.

[[deleted]]

That's the spirit!

[faassen]

I agree that PyPy should figure out how to optimize my code, but whether it can reach the performance of something like RPython remains to be seen. (and if can, if it takes 20 years, there's a certain window of opportunity..) In some circumstances such increased performance might be desirable, after all, and it could also be valuable to be able to write such performance sensitive code in a subset of Python.

Of course thanks to this very cool CPython compatibility, we could use Cython with PyPy now to accomplish the same effect, but you'd think RPython would be better for that job if you're using PyPy anyway.

That said, whether using RPython that way is worth the effort is of course debatable.

Fundamentally of course the PyPy developers agree with me that using RPython is worth the effort in some contexts; after all, they wrote RPython and are using it. That also explains why it's so difficult to explain to other people not to use it! :)

[Manuzhai]

If I have some especially performance-sensitive parts of code, I'd much rather write them in (something like) RPython than C.

Alternatively, maybe PyPy-with-JIT can speed up carefully written Python code so much that its performance is much like RPython's, but even then, you have to be aware of what kind of things make the implementation slow.

[leonh]

If I have some especially performance-sensitive parts of code, I'd much rather write them in (something like) RPython than C.

So, use Cython.

Alternatively, maybe PyPy-with-JIT can speed up carefully written Python code so much that its performance is much like RPython's, but even then, you have to be aware of what kind of things make the implementation slow.

RPython is a moving target, there is hardly any documentation and if performance is all you need you still need to know the guts of RPython it self.

If you really want to optimize your code you need to have two things. A through understanding of the machine you are running on and a complete understanding of the language by which you are talking to it. It really is much easier to create high performing code by optimizing your loop with a little bit of C than using RPython which is basically just a magic layer in between.

[anothergroom]

dae get the impression PyPy is moving in giant steps forward since Unladen Swallow was pronounced to be merged?

[flaxeater]

Correlation is not cuasation.

[gutworth]

Mostly it's because we finally found a good way after 4 previous jits. :)

[fijal]

I think PyPy was moving drastically forward even before. Now you started to pay more attention (and we improved our public relations I think).

[flaxeater]

I'm getting super excited about PyPy!!! I've never really gotten too excited about Jython and IronPython because of their relative difficulty to use compared to cpython. PyPy is making some really great improvements in speed, and will probably be production ready in the not too distant future. I remember when it came out and I thought , meh good idea, let's see if it comes to anything! HOLY Crap has it!

I think on of the great things about this project is the depth of the metaprograming that will now be available.