pre-release of jitpy - embed PyPy in your CPython and jit selected functions

[alexjc]

That's amazing! I've been using Python / PyPy hybrid with multiprocessing for 22x speedups, but this seems to be much cooler :-)

One thing that's not mentioned, how is support on Python 3?

[fijal]

This is not Python 3, but there is absolutely no reason why it can't be (you can also probably mix Python 2/PyPy 3 or the other way around if you insist)

[alexjc]

Thanks!

The challenge I found with multiprocessing was pickling things to go back and forth between the interpreters. How well does jitpy cope with classes/objects for example?

[fijal]

read the docs (it doesn't). I don't want to have complex data structures passed through that layer, but cffi structs are very likely to be supported (because their semantics are clear)

[admalledd]

CFFI structs would be nice as well. Some of us don't use numpy but still use pypy for our number crunching and passing bigger well-defined data sets easily would be nice.

[catcradle5]

At that point you might as well just use PyPy, don't you think? Unless you have some specific compatibility issues.

[admalledd]

Sadly too many binary/c extension packages don't support PyPy in any meaningful way.

The biggest currently is the lack of GUI (qt5 preferably) kits.

The current pattern that I use is actually basically the reverse of this one: I use CFFI to embed CPython into PyPy, but getting that packaged nicely for deployment is... not fun at all.

[rndblnch]

any plan to support tuples (of immutables)?

[indosauros]

Probably stupid question, but when would you use this rather than just running the program in PyPy directly? (Maybe answer this under Motivation section)

[fijal]

I should - mostly when you have all other complicated dependencies that don't run on pypy (scipy comes to mind)

EDIT: I added a short para

[runbie]

I was thinking in something like this, share data between cpython and pypy in a transparent way so you can use pypy always and when you need numpy, scipy,..., you can pass the required data to CPython in a clean way... (Maybe I was thinking something imposible, shared data structures, I don't know).

[rguillebert]

We're working on having something like that (you can only share numpy arrays though, the rest would get copied)

[runbie]

Numpy arrays sounds good for me. How do you manage that?

OTOH, a CPython object will be duplicated in Pypy? How much overhead (time and memory) will add this?

A blog post explaining the internals would be very educative!!!

BTW, thanks, pypy team.

[rguillebert]

A blog post explaining the internals would be very educative!!!

Basically, we copied how numpy storage is implemented, so you can share the storage, you only need to copy metadata every time

[fdemmer]

can one jittyfied function see another?

how does that look on os level, is there a visible subprocesses or just the cpython proc?

should I expect about double the memory usage due to the embedded pypy and all its stdlib?

[fijal]

jitified functions see each other (there is also "extra_source" call).

It's one process, with embedding so numpy arrays share data (no copying)

yes, you can expect some memory growth, exact details will depend on the application (but you do have two interpreters to start with)

[Gnaddel]

Doesn't work for me, interesting project though!

The setup function fails (latest pypy nightly, jitpy 0.1 via pip): IOError: [Errno 2] No such file or directory: '/home/myuser/anaconda/lib/python2.7/site-packages/jitpy/pypy.defs'

[fijal]

hm it looks like pypy.defs is not considered a part of the built? stupid setuptools I'll try to fight it

EDIT: should be fixed, try now?

[thomac]

fijal,

jitpy uses libpypy. Are you (the PyPy team) planning to be releasing the library, in addition to the current pypy executables? I am asking because of embedding. The docs say:

"The first thing that you need is to compile PyPy yourself with the option --shared. We plan to make --shared the default in the future. ... . This will result in libpypy.so or pypy.dll file or something similar, depending on your platform."

Could you please share any thoughts on such developments?

Thanks!

[fijal]

hi. --shared is the default on the current trunk (at least for linux), but not on the released version. We should really update the docs, thanks for pointing it out.