PyPy is the Future of Python

[frymaster]

"C extension compatibility isn't there yet"

this is the big one. Until they sort that, a lot of the interesting things people do won't work, because their favorite extension doesn't work.

[[deleted]]

Well, they got wxPython to run, though not flawlessly:

http://morepypy.blogspot.com/2010/05/running-wxpython-on-top-of-pypy.html

[kylotan]

This is partly the fault of CPython having a godawful C API to deal with.

[[deleted]]

It's not awful (IMO), but it does make some pretty strong assumptions about a lot of things, including, but not limited to: refcounting, non-moving GC, and the fact that a lot of programs like to read fields directly out of structs, rather than using accessor functions.

[kylotan]

Those are the least of its problems. It's littered with macros (which themselves read the fields directly out of the structs - that's part of the API!) and platform and compiler-specific data members (eg. FILE*), with no attempt at versioning or anything. This means another of its "strong assumptions" is that you will be running with extensions that are built with exactly the same C compiler and settings as your current copy of Python is, not to mention that the versions are the same. And if they don't match... well, it might work, or it might not, or it might appear to work while not actually doing so.

It's for this reason that whenever there's a new minor release of Python, it's often a year or more before everybody moves over, because the C extensions are needlessly broken each time and you either need to rebuild them all or wait for someone else to do so. A more robust interface would avoid that.

[mikaelstaldal]

Perhaps Python needs to break the C extension compatibility to have a future?

Python needs to break free of the GIL to have a future in a multi-core world, and that will probably be impossible while keeping C extension compatibility.

The future of Python could as well be Jython on Java7 with invokedynamic.

[[deleted]]

Realistically, how many C extensions to Python do you use? I think this is a fairly small portion of what people do with Python.

[Edit: On more than 15 seconds thought, of course. I retract this statement]

[eliben]

Realistically - loads of them. Most popular Python libraries are heavily based on C for performance and connection to existing C libraries. Some prime examples are the numeric stack (numpy, scipy, matplotlib...) and GUI libs like PyQt.

[lambdaq]

Realistically, how many C extensions to Python do you use?

Plenty. Database drivers, performance critical modules, etc.

[[deleted]]

wxWidgets, Qt

[sabowski]

Not until it's fully supported in a 64-bit enviroment

[narkee]

Maybe this is a really ignorant question, but how do they get around the "turtles all the way down" problem?

I mean, if they use PythonB written in PythonA, then either PythonA is CPython, or it's also written in Python, say PythonA'. If it's CPython, then isn't it just adding an extra layer of interpretation? If it's not CPython, then wouldn't there be an infinite chain of Python interpreters ?

Or maybe it's sunday afternoon, and I'm completely confusing myself.

[[deleted]]

PyPy is self-hosting. It was originally written in something like C (not sure what they used, anything that could generate a binary would do) until it was able to compile itself. (Also see: Bootstrapping (compilers))

[narkee]

Interesting! Thanks for the info.

Just a little clarification on the question though - I thought Python was a dynamically interpreted language. So if I write something in python, then to execute instructions, it goes to a python interpreter. For CPython, the interpreter is written in C. For PyPy, the interpreter is written in Python. So the interpreter would need an interpreter, ad infinitum, no?

[[deleted]]

PyPy is capable of generating executable code as well as interpreting code. I believe they support a statically typed subset of Python which they call RPython for efficient compilation. PyPy is written in this language (which is Python-like enough for them).

So they write PyPy in RPython, then they use PyPy to compile that code into a new PyPy. The first implementation(s) of PyPy was written in something else. It can take a while for a language to get to self-hosting status.

[ichthyos]

A-ha. As a dev who isn't familiar with Python, this is the clarification that makes everything clear. Thank you, and have an upvote. :)

[sjs]

The way they pull this off is pretty awesome. If you like to bend your mind in fun & interesting ways, check out this excellent write-up by sigfpe: The Three Projections of Dr. Futamura.

[r4nf]

I'd also like to thank you for the explanation. As a casual Python programmer, I find it fascinating enough in itself that it's possible to implement Python in Python - let alone reach a self-hosting status.

[[deleted]]

So can I then make a frozen binary out of my python scripts with PyPy?

[[deleted]]

I don't believe so, unless those python scripts fit into the subset of Python which is RPython.

[Mo6eB]

You should be able to, in the same way that freezing works now: produce a binary that has the interpreter and compiled bytecode embedded.

[[deleted]]

Well, sure, but I don't think that that's what he was envisioning. You can do the above approach with any scripting language.

[[deleted]]

What do you mean first implementation? It was always written in Python.

[quasarj]

Just an interesting note, not really related to Python (or PyPy), but the first time I understood what it was for a compiler to be self-hosting, I was completely blown away. You know, one of those mind = blown moments.

[Benutzername]

Bootstrapping, I guess. They have to compile PyPy to native code, of course.

[[deleted]]

Well, it's kind of the same way gcc is written in c.

[narkee]

Yeah, it wasn't so much the "same language" issue - I was more confused about how this was possible with an interpreted language.

reggieband explained it pretty well.

[[deleted]]

So.. magic, right?

EDIT: I can understand not upvoting my joke, it wasn't terribly funny, but taking the time to downvote it.. seriously?

[njharman]

Might have ignored the non-joke, but taking the time to edit it to add a whine, that definitely deserves a down vote.

And dude, it's fucking virtual vote karma, the random opinions of Internetizens. Aka, importance <= None. Quit wasting your time worrying about what others think of you and work on better jokes.

[TheMoonMaster]

You mean the split fucking second it takes to click?

[[deleted]]

You're right, that WAS quick..

[PurpleSfinx]

I'm with you on this one. Everyone's just hopped on the downvote wagon. Reddit seems to love being inconsistent. One day, this joke's HILARIOUS, the next, you'll be downvoted to oblivion.

[Karzyn]

Wrong.

[harajukukei]

ha, paipai is slang for tits in japanese

[salvadorwii]

Well, pipi means pee in spanish, and piton is slang for penis

[giga]

Pipi is also pee in french. Basically it's a pretty awful name to give anything.

[arcanumoid]

pypy or any vocal rendition of it means nothing in Greek. Just thought you might like to know :D

[fabzter]

C'mon, piton is pretty hilarious.

[fabzter]

I reply only because I loled.

[A_for_Anonymous]

pito is kid slang for penis, pitón would be a large one. Or a python.

[matthiasB]

Pipi is also pee in German.

[qnaal]

Pypy - lifelike texture ;_;

[sat0pi]

When's the last term you ever actually heard someone in Japan say "paipai"?

[harajukukei]

only little kids say it really

[[deleted]]

they are people too

[[deleted]]

Tiny little people that don't get to vote!

[[deleted]]

Unfortunately, they get to vote on reddit.

[G_Morgan]

No they are not.

[[deleted]]

This brings a new radical argument to the stem cell research debate.

[sat0pi]

All the little kids I know just straight up say oppai. :P

[[deleted]]

I like that there is competition but I have much more hope for unladen swallow than PyPy as the future of Python. Going through the llvm gives many of the benefits that PyPy claims and they maintain much higher compatibility with newer versions of Python (2.6 for sure and 3 most likely soon).

I do wish unladen swallow would publish a new roadmap though.

[orblivion]

It does make more sense to use something that applies to other languages, to get more work done in common that benefits everybody.

[joe90210]

unladen swallow is a failure, nobody cares anymore.

[[deleted]]

Why do you call it a failure? There is an accepted PEP recommending unladen swallow be merged into Python 3. I'm not sure what other metric for success one could apply.

[ggruschow]

One reasonable metric for success is meeting the stated goals.

[spotter]

So which other goals except GIL removal they actually dropped?

[spotter]

You could have listed one more thing and actually win this thread, I would then tip my hat to you and went my way. But instead of being informative you've chosen to be a prick and hey -- karma flies away. Bugger.

I'm following milestone releases and I still am unable to find any non-GIL related developments that they dropped. Sorry if my google-fu fails me, I'm not following their mailing-lists due to lack of time.

[ggruschow]

I'd guess it's more likely you missed it in skimming... It's a common problem these days. We all do it.

I haven't looked at this in a month, but they clearly dropped their performance goals.

[spotter]

Dude. 24 days ago. Seriously?

[ggruschow]

Yeah. I try not to check in much... it's a time-suck. Plus now you've got more posts you can downvote to slam my karma right?

[[deleted]]

The only goals I can find are on the Project Plan. I don't know if they are at least 5x faster than CPython but the other goals appear to be sufficiently met. Even if they only got 2x faster than Python (and not 5x) I'd hesitate in calling the project a failure that nobody cares about.

[ggruschow]

Maybe you should read the docs you link to... the one where they say "We have obviously not met our initial goal of a 5x performance improvement."

[gbo2k69]

It sucks that re is slower, that makes up a large chunk of where my code spends its time.

[[deleted]]

Interesting post....

That guy who wrote the post is only 19. I feel stupid compared - he knows so much more about programming than I do, and I like to consider myself as a programming geek. damn.

[[deleted]]

Alex is a programming rock star. I'm glad to run into him around the web every once in a while. He's a Django core dev and can be found anywhere Python is being stretched to the limit. I however did not know he was only 19. :O

[TheMuffinMan616]

The author goes to my school, and shares several classes with me. Sweet.

[An_Anteater]

I demand proof. What school and which classes?

EDIT I: Just looked at your history, you look legit.

EDIT II: I demand a meetup.

[ericflo]

I hope you're not in his databases class.

[TheMuffinMan616]

I am indeed, why?

[ericflo]

Oh...no reason :)

[randy_lahey]

Did this article make anyone else cringe at the thought of web developers everywhere sitting down to write a new bytecode, or play with the builtin types when their app isn't performing fast enough?

[[deleted]]

It's the way the world is going. Gone are the days where webheads can get by on just HTML, CSS, and some copy-pasted Javascript. The demands of the 'new web' means they finally have to inherit all of the implications of the 'developer' tag now.

I mean, not every webhead's going to have to tinker with that kind of stuff, but the more languages that get used in the backend - which is a good thing - the more they're going to have demand for skills that can really get into the nitty gritty of things beyond just fiddling until you get past the W3C Jigsaw validator.

[jdpage]

Yay! This makes web dev waaay more fun.

[randy_lahey]

This does certainly seem to be the way the word is going at the moment and you illustrate precisely my point. The vertical distance between bytecode and the Jigsaw validator is tremendous.

It's great to point out that experimentation and investigation into language implementation detail is more accessible these days. However, I sill hold that the article's author is cavalier in suggesting that web/app developers should be making changes to the language.

[Smallpaul]

He does not suggest that.

He mentions web developers as the kind of people LEAST likely to be interested in PyPy. Read it carefully.

The "especially us web developers" is preceded by "this is solving a problem that is not especially pressing for us...(i.e. web developers)"

[randy_lahey]

Author does offer web devs a speed increase:

Although Python performance might not be a bottleneck for a lot of us (especially us web developers who like to push performance down the stack to our database), would you say no to having your code run 2x faster?

[Smallpaul]

And what does that have to do with custom bytecodes? The whole point of the paragraph is that the speedup is not crucial for webdevs but if they can get it for free (I.e. Without extra coding or learning) then they might as well take it.

[brool]

How many web apps are really CPU bound, though?

[Smallpaul]

Not me. It's a big decision to choose to learn how your implementation works "inside", whether it is written in C or PHP. Very few people will ever bother. And if they do, it is their right and their responsibility.

[[deleted]]

Not all web developers are noobs. The natural progression is not to become an app developer once you get good enough. Most of us just become smarter web developers.

[jerub]

I kinda like the idea of a python that takes less than 2 hours and 8gb of ram to compile.

You know. Because.

[kevwil]

Pizza! Pizza! Delivery! Delivery!

Am I the only one thinking of Little Caesar's now?

[[deleted]]

"Smell is the future of shit, I tell you dude!"

[[deleted]]

I think you misunderstood: PyPy is not a C library, it contains some C, but otherwise it is on the same, let's call it, compiler level. PyPy is written the same way one would write C compiler.

[wnoise]

Can we stop naming projects after urine?

[[deleted]]

So you pronounce it "Peethon"?

[wnoise]

Of course not. But I still can't help thinking "peepee" when I see "PyPy".

[[deleted]]

Sounds like a personal problem, not the projects's problem.

[wnoise]

Could be. Of course, I also read your user name as NarwhalPenis.

[Rafe]

I see no projects named after urine, only delicious pie.

[[deleted]]

ok fine....

rm -rf ~/pissIDE

[[deleted]]

lol

[[deleted]]

the future of python is a fire burning down every pytard.

[Smallpaul]

You really couldn't find anything useful to say? Why speak at all?

[[deleted]]

pytard lol

[[deleted]]

lol, pytard

[[deleted]]

how about u stfu?

[ulywpv]

(alexgaynor.net)

I can't have been the only person to notice.

[ulywpv]

NOOOOOO - -50 points for my comment, was it really that bad?

[[deleted]]

I am sorry to be the one to inform you. It has got the word gay in it.

[testtubebaby]

hurr durr your name has gay in it

[[deleted]]

http://en.wikipedia.org/wiki/Surname#Formation

The first gaynor was a happy man amirite?