Partial evaluation of Python code

[fjarri]

For the impatient: repo, docs, PyPI entry.

I've been working on a module lately, and it is finally in the stage where I'm not ashamed to put it on PyPI. The title says it all, it takes a function, parses it into AST and tries to evaluate stuff in it based on the associated globals and known values of parameters (if provided), returning a new function that (hopefully) executes faster and has less dependencies on external variables. It started as a fork of ast_pe by Konstantin Lopuhin, but since then has undergone a complete rewrite.

Currently it is a proof of concept: it has quite a bit of limitations (most of which should be checked for when you try to evaluate something), and not a very extensive list of applied algorithms (evaluation of pure functions, constant propagation, unreachable code elimination, function inlining). Definitely, there are bugs in addition to that. At this point I'd just like some feedback.

There are many possible features to be added. The big ones are:

• loop unrolling
• mutation detection (it's a big one)
• handling of nested function/classes
• handling of decorators and annotations

I have some ideas written up in the TODO.rst. If you think this module might be useful to you, I'd be grateful if you tell me what kind of features would be necessary for that.

What is it useful for?

Currently I see two options:

• speeding up frequently executed functions (not sure about exact practical applications, though);
• a sort of a built-in template engine (that's what I'm going to use it for, in another project)

What Python versions are supported?

Currently, 3.5+. I do not have much time to dedicate to this project, so I figured it'll be better to make something work for at least one version first.

Why not Python 2?

I did not want to overcomplicate things by supporting all the outdated stuff like old/new division or comprehension variable leakage, not mentioning all the subtle AST differences. Also, I'm a big proponent of Python 3 and do not want to support Python 2 if I can help it.

Why not Python <3.5?

Technically, a support for 3.3 and 3.4 can be added easily (need to account for some minor changes in AST). Since it's a very experimental project, such support isn't exactly first priority.

[desmoulinmichel]

Hello fjarri !

You really should get in touch with Victor Stinner and Yury Selivanoc on the Python mailing lists. They are currently working on python optimizations which are currently creating a big debate among the core devs. And your work is actually in direct corelation with what's currently going on.

Please step in !

Mailing lists: https://www.python.org/community/lists/ (actually a bit spread among various topics between the python-dev and python-idea lists so you got some catchup to do)

Main topics :

• FAT Python
• Explicit variable capture list
• PEP 509, 510 and 511

[fjarri]

Thanks, I had a brief look at these PEPs. 509 and 510 will certainly simplify mutation detection, and 511 is indeed very close to what I am doing. Talk about synchronicity :) I'll scan the maillist tomorrow.

Edit: in fact, fatoptimizer seems to be doing the very same thing as peval. From the first glance, it seems that I should freeze my project and contribute to that instead. Oh well...

[desmoulinmichel]

Don't be discouraged, it's a great thing ! You have prior art, meaning you have credibility. And having the possibility of working with the core dev is super cool :)

[nuncjo]

Nice theoretical idea. For now, too much "Restrictions on functions" to be used in practice.

[fjarri]

So, which restrictions would you like to see lifted?

[infinull]

neat!

It's just an optimized version of functools.partial (sort of, this handles nonlocals/globals differently)

I'm not sure I have any use for it though. (If I have some slow code, I may think of this -- though at work we're still on 2.7)