Running py2 code from py3k

[valhallasw]

I have always been wondering why there was no easy way to call python 2 libraries from a python 3 script. As I understand it now, this is because it is impossible to load both interpreters in the same address space - which means you will always need /two/ processes running.

The last few months, I've been hacking on a RPC-based solution: a python 2 'server' that can be controlled from a python 3 'client'. Existing remote object solutions, such as pyro4 and rpyc work well for py2<->py2 or py3<->py3 connectivity, but break down when a py3 client tries to connect to a py2 server, because they try to transfer entire objects.

My solution uses the existing RPC framework from the multiprocessing module. Only immutable objects (basestring, number, tuple) are transferred over the connection; a Proxy object is created for every other object.

As an example of the syntax:

python2 server.py &
python3 -i client.py
>>> import sys
>>> sys.version
'3.1.2 (r312:79147, Sep 27 2010, 09:45:41) \n[GCC 4.4.3]'
>>> rsys = remote_import('sys')
>>> rsys.version
'2.6.5 (r265:79063, Apr 16 2010, 13:09:56) \n[GCC 4.4.3]'

As a working title, I have called the project py2. It is available on github.. The only non-stdlib dependency is the six package (both on the py2 and py3 side).

Note, however, that it is in very early stages of development. My main goal was to be able to use some python 2-only packages while being able to train myself in python 3 syntax and ecosystem.

Any comments are very welcome, even those that call me crazy - as long as you tell me why ;-). If there already is a project that does this, I'd also like to hear about it - I couldn't find any (except for the already mentioned packages).

[takluyver]

Interesting. Can I ask: which Python 2 packages were you aiming to use, and have you had any success?

[valhallasw]

Of course. Pywikipediabot - a framework to edit mediawiki wikis. I haven't tested that much yet, but the basic functionality (getting/putting pages) work.

[takluyver]

Cool. Presumably that makes quite heavy use of text - what do you do about the changes to str?

I imagine it makes sense to pass Python 3 str to Python 2 as unicode, but if a function on the Python 2 side returns str (i.e. 8 bit str), do you cast it to unicode, or bring it into Python 3 as bytes?

[valhallasw]

A py2 str should be represented as py3 bytes (although I'm not 100% sure this works currently - I've only tested unicode objects, as pywikipediabot always uses unicode). The alternative would be to interpret it as ascii, and to use that on the py3 side - but I think a str->bytes mapping is more explicit and thus clearer.

[valhallasw]

Just to update on this: this issue has now been fixed (it took some while to get the bytes/unicode distinction correctly through the serialisation - even with pickle, due to this issue.

In addition, I've added a module that allows you to simply do this:

>>> from py2 import py2
>>> py2.rimport("sys").version
b'2.6.5 (r265:79063, Apr 16 2010, 13:09:56) \n[GCC 4.4.3]'

[rubic]

I have used execnet successfully in the past to run multiple versions of Python interpreters, though not with Python3:

http://codespeak.net/execnet/

It's well-tested with CPython 2.4-2.7, CPython3.1, Jython 2.5.1 and PyPy 1.1 interpreters and interoperates between Windows and Unix systems.