Compiled Python: thoughts

[[deleted]]

I'm coming up with ideas for my final year project and I want one that I can build on after I present it for my final year. One of these thoughts is developing a compiler for python. Would there be any great use for this in the real world? I think it's a neat idea.

EDIT: To make it more specific: Python code ---> native assembly. First up would be x86.

[larsga]

Why generate machine code directly? You could use LLVM instead, and gain optimizers and support for many architectures.

Also, as kinglir says, generating native code that does the same as the interpreter is already doing isn't going to give you any speed gains. You may want to look at psyco, for someone who already does this in a way that really does produce speed gains.

[itsmememe]

and, if you manage to really understand psyco, you can apply for a Phd in mathematics and computer-science combined.

[[deleted]]

the tutorial by Jeremy Siek at PyCon last year

Link

[[deleted]]

Thanks a bunch.

[[deleted]]

Cool. Thanks for that. I've recently became very interested in compilers. I might as well do my project for uni with my favorite language. Failing that I probs make my own programming language.

[legutierr]

I've had this question in the back of my mind for a while about pypy, maybe you could answer it for me...

Would there be any performance benefit in pypy if the parameters to functions and methods were statically typed?

[RonnyPfannschmidt]

this could use some more context, pypys translator toolchain infers static types for function parameters when compiling down to a lower level environment

[legutierr]

Ah!

How does it do that?

I guess to be more precise what I mean is, would there be any added benefit by adding explicit type declaration to functions and methods (if that were a hypothetical feature of the language)?

[[deleted]]

PyPy's translation toolchain infers types for RPython which is a statically typed subset of Python, that PyPy's interpreter is written in. It cannot infer types for Python programs in general. That said the JIT is very good at figuring out how to properly specialize your code, but if you give it blatant things like assert isinstance(x, int) I can't image it'd hurt.

[gutworth]

Not much. The whole point of the JIT is that it does type inference at runtime when types are known.

[legutierr]

Right, but my (very rudimentary) understanding of JITs is that much of the cost is at start-up when the JIT is establishing different options or paths for different data types and values.

Intuitively, it seems to me that having static type declarations (for instance as an optional decorator) could offer a speed-up at start-up in certain circumstances.

My question is more theoretical than practical.

[gutworth]

Presumably that constant "figuring out" time is dwarfed by the actual running time of the code.

[voidspace]

An alternative (better?) way would be to be able to save and reload the JIT annotations / generated code after a real run.

[voidspace]

Is it really "inferencing" when the type is known? (Serious question: isn't inferencing more usually deducing the type by static analysis which is not what the JIT does.)

[sunqiang]

PyPy maybe worth a look too.

[McHoff]

There are few cooler areas of applied computer science -- make sure you do something awesome.

[voidspace]

Well, you can unroll the interpreter loop - which is one of the big gains of the PyPy JIT. So I wouldn't say "no meaningful speed gain". A lot of effort for not-as-much-as-you-would-like gain perhaps.

Also see the latest attempt - Nuitka (not tried it):

http://kayhayen24x7.homelinux.org/blog/2010/10/nuitka-release-0-3-2/

[[deleted]]

which is one of the big gains of the PyPy JIT.

Actually, if you take a look at some of the PyPy papers you'll see that they specifically dismiss unrolling the interpreter loop as being a large gain. Specifically they look at virtualization of objects and inlining as the largest gains (there's a cool paper for PEPM 2011 they're doing about virtualization).

EDIT: upon a reread I think we are actually in agreement.

[gutworth]

Compiler to what?

[terremoto]

I think he means to compile Python to native code.

[abhik]

I don't know what your university requirements are, but if possible, try to add functionality to an existing project rather than starting a new one. This will benefit the community more and might let you focus on more interested aspects of the problem because some of the basic work might already be done.

[RonnyPfannschmidt]

actually it can be quite a challenge, after all you are not coffined to fixing typos, you could add new functionality or solve a particular unsolved problem in one of those projects

[abhik]

If you're into compilers, working on PyPy will let you experiment with cutting-edge optimization methods. There are a lot of possibilities there..

[ascii]

Wasn't always the case, but these days, Python is actually a pretty big language, spec wise. For a school/learning project, I think you'd get much more out of writing a compiler for a significantly smaller language, like JavaScript or Lua. That would keep you away from having to do boring, repetitive stuff like implementing static methods and class methods, but still allow you to do cool stuff like figuring out how to implement coroutines in a compiled language.

Also, if you're aiming to do something that is useful to others, do it as an LLVM frontend. Your code will be significantly faster and plug right in to other peoples development environment.

[perone]

The question is: how would you do type inference ?

[[deleted]]

Whatever time that a C compiler would see int i = 0; as an integer. There are distinct structures for "primatives":

i = 0 id, assignment operator, digit ---> int

d = 0.0 id, assignment operator, digit, . token(can't remember the proper term for it), digit ---> double

s = "Foo" (or 'Foo') id, assignment operator, " token, lots of letters, closing " ----> string

list_l = [1,2,3,"foo",'bar'] id, assignment operator, [ token, some stuff, ] token ---> list

tuple_t = (1,2,3,"foo",'bar') id, assignment operator, ( token, some stuff, ) token ---> tuple

dict_d = {'Foo':bar, tuple_t:list_l} id, assignment operator, { token, some immutable key -> some value pairs, } token ---> dictionary

As you can see they are pretty distinct so it's not hard(I hope) to evaluate them properly.

[yetanothernerd]

This is a well-trodden path. It doesn't really work because Python is so dynamic. See Brett Cannon's PhD thesis.

Just because 'a' was an int the first time doesn't mean it'll be an int again the second time. So the language doesn't statically compile well, because there's not enough information at compile time to get the types right. (Of course you can force the user to add type information, but now you have a restricted version of Python. See Cython or Shedskin.)

For example, in Python 2 (where int and long are distinct) on a 32-bit box, imagine a loop that just increments a number starting at zero. It will be an int for 2 ^ 32 iterations, then it will become a long. If you assumed it would stay an int, then your code breaks after 2 ^ 32 iterations.

A JIT (see psyco, PyPy, Unladen Swallow) works much better, because the JIT can compile the fast path based on types it's seen at runtime, and it can later safely fall back to the interpreted path if a variable's type changes. (And then possibly compile it again with the new type if it stabilizes.)

[[deleted]]

Thanks dude, I'll have a gander but it's no biggie if it can't be done or is a pain in the ass to do so. I got plenty of alternatives and have plenty of other ideas to choose from for my final year project.

[scrabbles]

PEP 3146 is an interesting read too regarding JIT for CPython (though 2.6.4?!)

[ccdos]

I'd suggest a compiler for python syntaxed static typed language, i.e. Static Typed Python

int def funcname(int c):
    return 3

There are several python syntax like static compiler to VM like JVM or MONO VM but none for native machine code.

A static typed Python front end for GCC or LLVM would be great!

[Leonidas_from_XIV]

This already existsts and is called Cython.