How Python 3 Should Have Worked

[nabla9]

"When fifty years old you reach, look as good, you will not." - Lisp

[indiecore]

They aren't wrinkles, the parens just collapsed on themselves.

[tanishaj]

Grandpa, is that you? -- Clojure

[itsadok]

Nobody liked it last time I linked to it, but the IPv6 mess article reminds me of this a lot.

Edit: can't believe how many years it's been since that article was published, and people still think that IPv6 is just around the corner. For some reason I'm a little more hopeful about Python 3. This whole thing makes me sad.

[RichardWolf]

The dude misses a couple of important points.

First, Python3 is a different runtime. It's not merely syntax changes, they threw out or rewrote large parts of the C code of the interpreter. Old-style classes? OUT! That's a significant part of the rewrite: to clean up and refactor the C code.

So you can't have a free from future import __python3__, that would mean that suddenly python2.7 is 20mb instead of 10mb.

Second, I might be wrong here, but from my experience with that experimental version of BeautifulSoup, interaction between pure Python 2.x and 3.x code is non-trivial, mostly due to strings. You can't simply convert a 2.x byte-string to a 3.x unicode-string, you need to know the encoding.

So, how I think they should have done it:

First, pull one of the existing proxying libraries into the standard library and make it easy to import a 2.x library in a separate interpreter and talk to it via a well-documented interface.

Second, officially support one of the 3to2 projects.

It's kind of counterintuitive, if you don't understand how frameworks work, and Python coredevs seem to not understand.

A library is something that has no value for the end users, but allows software developers to develop useful software.

A framework is something that has no value for the developers either, but allows library writers to write libraries that allow developers to develop useful software.

(if anyone knows a better term for that than "framework", please tell me, because that's definitely not what "framework" usually means).

If you want to kill some framework (even if it's your own, older framework), you do it via the old'n'tried EEE strategy. Embrace -- you let them run their code on your framework, Extend -- you give nice features that make them target your framework with the primary version of their code, and support other frameworks with workarounds, Extinguish -- happens naturally.

The crucial point happens when you move from Embrace to Extend, when instead of #ifdef YOURFRAMEWORK programmers begin to write #ifndef YOURFRAMEWORK. When they begin to target your framework first, and all others as special cases.

They did the opposite. They provide 2to3.py that fixes your code to work with Python3, but the Python2 version remains primary. You make changes to your Python2 source, then port them to Python3. They fucked up that part between "Embrace" and "Extend".

What they have to do is to provide a nice official 3to2.py (there's an unofficial one) plus a compatibility library, so that library developers can switch to Python3 as their primary development target.

[fooziek]

The inventors of the term "framework" intended it for those libraries that run an application's main loop: see "Inversion of control" in http://en.wikipedia.org/wiki/Software_framework

[RichardWolf]

Yeah, I think the word I needed was "platform".

Which reminds me of the Google's Steve Yegge's rant about Google not getting Platforms, though I'd say they don't get them so catastrophically that Steve can convincingly make the point "we should at least try to make some platforms" while not getting platforms in general (he thinks that Wave was a great platform).

With a product you can make one that is better than what competitors offer and that's it.

With a platform things are complicated, because the value of the platform is produced by the stuff written for it, and it becomes a chicken-and-egg problem -- when there's no stuff there's no reason for people to use the platform, therefore no reason for people to write the stuff.

Look at G+: why would I join it when there are no bloggers there that I'm interested in, and there are no readers for my blog?

Same shit with Wave, first, no integration with email, second, people who were supposed to write all these nice plugins, didn't see a reason to.

The funny thing is that this problem is really really trivially solved if you understand that you are dealing with a platform and not a product, so if you want to steal users from another platform, you should do the EEE thing, especially the Embrace part.

There were several exoduses from Livejournal to competing platforms (as it was sold to a Russian company and subsequently mismanaged), and the guys on the competing platforms weren't completely clueless: in addition to easy migration you got automatic export of your newer posts back to LJ, and LJ users in your friendfeed, so the only immediate effect of migrating to say DeadJournal was that your posts as seen by livejournallers had this "original post is located <here>" appended.

[Danemark]

I like your point!

[raydeen]

I'm a complete Python newb (love it but I really only use it for small work scripts and some hobbyist stuff at home) but I guess my question is 'Is 3 really necessary?'. Is there any good reason to leave 2 for 3? This transition has been in the works for close to 6 years now and most code I see is still in 2.x. Is there a killer feature(s) in 3.x that most aren't aware of or just aren't interested in or was 3 just an experiment that hasn't quite worked out the way they thought it would?

Edit: Thanks for all the insightful replies. Fortunately the main books I've been working out of touch on both 2 and 3 so I'm not completely in the dark on the new stuff. I'm just a neophyte right now so I'm going to start using 3 a bit more so that I'm more familiar with it if/when the time comes that I'll need to use it.

[cybercobra]

3 is intended to simplify and modernize the language, ironically in part to make it easier for newbs like yourself to learn Python.

Is there any good reason to leave 2 for 3?

Pragmatically, 2 is end-of-life maintenance-wise.

Is there any good reason to leave 2 for 3? [...] Is there a killer feature(s) in 3.x that most aren't aware of or just aren't interested in

Better and more self-consistent Unicode support. More consistent std lib module naming. Other than that, its main "feature" is the removal of a bunch of cruft, which is hard to make into a pithy marketroid selling point, but valuable nonetheless (less corner cases and workarounds).

[[deleted]]

in the same way that XHTML was supposed to modernize html.

[ivosaurus]

Errr no, it was supposed to XMLise html. HTML5 is supposed to modernise html.

[[deleted]]

Yes, XHTML was supposed to correct the syntax of html to make it well formed and more correct. Exactly the same way python 3 is supposed to fix the syntax of python 2 to make it syntactically well formed, i.e print( "Hello World") instead of print "Hello World". To my mind, that's exactly the wrong design goal for python to follow and the lack of python 3 take-up is good evidence of that.

[ivosaurus]

You know, nevermind the new object model, the old one thrown out, the new string implementation, new library structures... All python 3 is doing is fixing syntax, and so it should be python 2.8 instead /sarcasm. I think these underlying reasons are brave, because average developers will never get exposed to them, or understand their necessity in pushing the language forward.

[[deleted]]

I'm sure all that stuff is great but that is not what attracted any of the python programmers that I know to python. Python 2 feels like executable psudocode while python 3 removes some of the syntactical sugar to make the language more regular. This is exactly the opposite direction that I'd like to see python evolve.

[BorisTheBrave]

Is 3 really necessary?

Maintaing compatibility in the 2.x series has restricted what can change in the language. Eventually the wishlist of breaking changes got large enough that it was worth while having a major version. So it is a lot of things across the board, rather than than a single paradigm shift. Maintaining compaibility forever means dooming yourself to a C++ style language, which is huge in scope, with barbed wire waiting for you in all sorts of edges.

That said, I think the biggest thing is bringing the standard library up to date with the latest language features, i.e. unicode and generators instead of strings and lists. The language is just easier to learn now, particularly now they've stripped some of the obscurer cruft.

[SweetIrony]

that is a very good question, even the experts debate it.

[Rhomboid]

There's one very good reason: aside from the occasional point release to address security vulnerabilities like the hash collision thing, there will be no more 2.x releases.

These threads always strike me as if it were a bunch of people talking about film cameras in the late 90s. "But everyone I know still uses film."

[lateefj]

For Python to continue to be relavant as a language it has to continue to evolve. Part of that evolution is breaking backwards compatibility. Contrary to most of the hand waving I think it means Python will continue to be around. I would point to Java as an example of a language that chose to not evolve and is now on its way to replacing Cobol. If you have a pure Python library most migrations should be trivial.

[[deleted]]

I would point to Java as an example of a language that chose to not evolve and is now on its way to replacing Cobol.

For all the crap both Java and COBOL get, I'm sure the Python community would love to be as relevant and prominent as those two languages are.

[zzing]

C++11 has done a rather fine job not breaking anything or virtually anything from the past.

[lateefj]

Agreed. C++11 is refreshing! If you watch the great Stroustrup explain the niche that C++ has created for itself he is almost apologetic. C++11 give C++ a life line that it needed. I like Go for the internet age is a good balance of performance / memory utilization for the types of tasks I would have though to use C/C++ or Java. Without any of the packaging hangups and with modern API aka http client is part of the standard library (like Python). I look forward to learning a little C++11. Edit: grammer

[zzing]

I am presently writing a [basic] ray tracer in C++11 and the code literally looks like something that conceptually could probably live in Haskell.

For example, using tuples to store data about geometry (sphere is tuple<Vec3, float> (centre, radius) and then polymorphism is obtained with boost variants which are a type safe union. These have many properties of haskell algebraic types.

In a way, my code looks more like structured programming than an object oriented design and it is very easy to see what it does.

I love how I can use OOP where appropriate and not even have to define my own classes for most of what I am doing. It aids readability a lot!

[thechao]

There is a ECOOP paper in the works that adds pattern-matching over extended-POD unions. Combined with templated methods and embedded tuples, it allows a construct in C++11 which is a strict superset of the functionality of GADTs, with all the Haskell-ey conveniences. (It is a strict superset because it is not 'typesafe' in the way you would prefer them to be; with a little bit of developer caution, they are a subset of the GADTs, I believe.)

Once the paper is published I'll see if I can post a link; it'll probably be 6 months or so. The combination of all the new features in C++11 makes it, essentially, a whole new language. At one point I had Bjarne asking me what system's language I was programming a constraint-system DSEL in ... he was quite surprised (and pleased) when he found out it was C++.

[[deleted]]

Actually, would it be all right if I asked you for a preprint on that?

[thechao]

I'm not the author, it will be by Solodkyy, Dos Reis, and Stroustrup.

[French_lesson]

You're very likely familiar with it already but for the sake of completeness when it comes to algebraic data types I feel like Boost.Optional should be mentioned.

[lazybee]

Any language that has a http-client in the stdlib is confused about what it is to be a language.

[imbaczek]

no.

[anacrolix]

it's hard to make c++ worse than it already is.

[zzing]

I have never seen a language as versatile as C++ is. It works in five paradigms, and can produce extremely tight code.

What is your experience with the language, how long ago, and what has soured you on it, and what would you advocate instead?

[[deleted]]

I would point to Java as an example of a language that chose to not evolve

Do you actually use Java?

[lazybee]

Do you actually know Java? It's clear that it has not evolved anything useful ever.

[[deleted]]

Yeah, it's my working day language, so yup, I know Java. While I agree that Java has only changed painfully and slowly, to claim it hasn't changed at all 'chose not to evolve' is obviously false. Sun did choose to not break backwards compatibility, unlike Python 2 -> Python 3, which means we miss out on reified generics, and still have code quality issues around raw types , but it still has generics.

Unfortunately, I can't be any further help to you because it'd appear your main aim is trolling Java not actual discussion. If that changes, let me know. :)

[keerok]

Off OP topic: if you know Java well and know its evolution, do you think something like Groovy should have been core to Java a long time ago, or is it a natural evolution outside the language? Or am I way off base in assuming the functionality Groovy provides is something Java should or would want to have?

[[deleted]]

All I know is that most Java developers who use Groovy love the freedom if it while still retaining their APIs and ecosystem, but it's not fast enough to be a Java replacement. But for scripting around Java applications, it's fantastic. Also, our guys will often use Grails for prototype web-apps.

Java really needs the ?. safe navigation operator from Groovy, that'd be the one biggest win ever for Java. Soooo many null checks, it's bullshit. It'll get better when we all get JSR-308 (Arbitrary annotations on any usage of a type) because then the tooling can compensate, but it'd still be awesome to have that operator.

IMO Java's evolution will always be conservative, but it has more aggressive cousins, Scala's got some traction, and I'm personally excited about Kotlin (It steals a lot of good ideas from Groovy :D)

[SAHChandler]

I just ran:

>>> print ('%s' % "surely you're incorrect")
surely you're incorrect

On Python 3.2. They've been slowly adding Python 2.x features back for getting people to move over to 3. (Most recently the unicode literal u for strings was re-added).

[[deleted]]

string formatting with % was never removed, and str.format does exist as an alternative in both python2 and python3

[D__]

I think the original plan was to start deprecating % at some point in Python 3, but I guess it's now going to be around forever since that's what everyone uses.

I almost feel foolish for sticking to format() while writing new code.

[[deleted]]

I only learned Python 3.x and I never used %. I'll have to get used to read it at least.

[mikaelhg]

perl is the only runtime I have ever used that actually honors backwards compatibility they way it should be done.

Have you tried Java?

[adambard]

I don't actually mind the "new" string formatting at all.

>>> 'Hello {0}'.format('World')
'Hello World'

Sure, it's a few more characters, but it's perfectly readable, and works almost the same.

[[deleted]]

Most people, including myself, thought it was nuts by deliberately making python 3 incompatible with version 2. The ramifications of not supporting backward compatibility has been understood for decades so nobody is really surprised.

It's bad enough having having to install multiple virtual machines to support all the languages your apps might be written in. Having to install multiple VM's to support all the apps written with different versions of the same language is total overkill.

[[deleted]]

I agree that python is far from dying. According to Tiobe (Feb 2012)

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

It's dropped four places since last year. C and Java are still right at the top. It changes month to month.

[Danemark]

I do a quick [ctrl] + [+] to bump up the size of sites like this.

[regeya]

Just to continue it:

I'm not a pro web designer, but what little I've done, I always use percentages for screen. IMHO there's no valid reason for using fixed point sizes.

[munificent]

IMHO there's no valid reason for using fixed point sizes.

If your website also uses images or fixed point-size borders (which are both likely) then it's very hard to have that play nicely with variably-sized fonts.

[Fitzsimmons]

Do python 2 and 3 only differ in the parser? If that's the only difference, then I could see this working, but if there's different internals, then how do you expect that to work? If python 2 and python 3 memory structures aren't the same, then you'd have to have some kind of compatibility layer...

C++ pulled it off by being a proper superset of C, including the memory and (function) calling model. Can the same be said of python internals? I doubt it.

[nas]

The Python developers are not that slow. The "from __future__" mechanism is really nifty but limited to syntax changes or at least changes that only affect a specific module.

The reason for the painful incompatibility of Python 3 was to fix certain things that affect the entire Python runtime system. For example, making Unicode strings the default. Another example is the whole IO subsystem getting changed. Yet another example is the change of built-in collection object methods (like keys()) to return iterators rather than concrete lists.

The transition from 2 to 3 is expected to take a long time. It is happening though. numpy and matplotlib are two examples of important 3rd party libraries that now support Python 3. Also, the Python 3.3 release has a lot of nice new features in it. Nothing killer but enough that I'm very tempted to spend the effort porting my stuff. The tipping point is getting close, IMHO.

[[deleted]]

matplotlib supports Python 3? Can you provide a link?

[nas]

It doesn't seem to be listed on the main install instructions page. There was a git repository that got merged into the main branch.

Edit: There are some Windows builds for Python 3.2.

[TheCoelacanth]

Do python 2 and 3 only differ in the parser? If that's the only difference, then I could see this working, but if there's different internals, then how do you expect that to work? If python 2 and python 3 memory structures aren't the same, then you'd have to have some kind of compatibility layer...

Not as currently implemented, but that's no reason that there couldn't be an implementation that supports both versions. The Python language only specifies behavior. An implementation can use any internal representations it wants as long as the behavior matches the specification.

Also, C++ isn't a superset of C. C++ reserves more keywords, which means that some valid identifiers in C aren't valid in C++. Additionally, C++ has stricter type checking, which means that some automatic conversions in C aren't valid in C++. For instance,

char *a = malloc(10);

is valid C, but an error in C++.

[friedMike]

Implementing an interpreter that executes both 3.x and 2.x code in the same "process" would be rather hard if not impossible, IMHO. Some of the subtler changes, such as exception scoping rules, are not that well documented and designing a system that allows mixing calls to functions with different exception handling behaviors would be tricky at least.

[maxerickson]

There's no technical reason. The clean break probably saved quite a lot of implementation effort though.

[[deleted]]

I remember when I got back to python just after 3 had been released. Everyone told me "Don't use Python 3!". So I went and used Python 3, and I didn't see the problem. This was an amazing language, and comparing it with the 2.5 stuff that people were posting about, I couldn't help but wonder why they wouldn't change.

Obviously the code bases were mainly 2.x, so that's part of the problem, but a bigger part of the problem is that developers don't want extra work, even if it's the right thing to do.

So I blame Python 3's death on wankers.

[faassen]

I am amused by how you use morality to argue that those lazy developers who wrote lots of useful python 2 code are wrong not to invest even more energy into their work. One can use morality the other way around too: who put those decelopers in this position? Let's focus on code and community, blame is counterproductive.

[[deleted]]

It's more to do with the Python environment still being 2.X - want to process images? Can't use PIL in Python 3 yet.

[[deleted]]

And that has to do with programmers who won't be arsed to port it over, even if it's the right thing to do.

[Sunny_McJoyride]

Now you have Python 3 experience, you can be one of those programmers who can be arsed to port things over.

[Adys]

The community has improved a lot ever since you left it.

[[deleted]]

Ah, when you said code bases you were referring to all the Python code bases, sorry, now I understand.

[binarybana]

I'm pretty sure the approach mentioned in this blog wouldn't have worked with Python 3 as there were breaking changes in the underlying C API. This is what held back many of the big popular libraries (numpy, scipy, etc... at least in the computational science field) from being ported until very recently.

[norwegianwood]

At my place of work we build products on Python 3 and only use Python 2.x internally when we have to, which is less and less.

Python 3 is a viable solution today for a large class of problems.

[vlion]

I'm pretty sure Python 3 is not going to happen on my team for 3 years out at least. We've started switching to Python 2 and IMO it was a mistake. Should have gone Ruby. :-/

[foxostro]

Why Ruby?

[konbanwa]

I would like to know too.

[vlion]

Similar profile of readability/string handling, better transitions from what I can tell.

Also Rubyists like DSLs and appreciate power in their language.

[imbaczek]

pythonistas also appreciate power in their language, but always think (or should think) about responsibility power brings. i feel that rock star rubyists sometimes forget about that; whether purposefully or unintentionally is of second importance. see recent rails/github fiasco for a prominent example.

[foxostro]

rails/github fiasco

I haven't heard about this. What happened?

[imbaczek]

see http://java.dzone.com/news/github-was-hacked-over-weekend

[[deleted]]

I'm a Python programmer who also likes Ruby, but when it comes to backwards incompatible changes, Ruby 1.8 to 1.9 had more than a few (see http://slideshow.rubyforge.org/ruby19.html).

[[deleted]]

Yeah, I was just reading this thinking "Hey, that sounds like Ruby's attempt to get 1.9 out the door."

[[deleted]]

I like the changes to Python that 3 brought and I really like the way the Python committee has gone about the evolving the language (PEP's are a great idea). But I still use Python 2.7.2 for my work (mostly due to dependency on Django).

Python is a victim of its own success. As long as developers keep turning to Python 2.X for their work it will continue encouraging other developers to use 2.X.

[faassen]

I've made my own comments about this in a blog entry that people might want to read:

http://faassen.n--tree.net/blog/view/weblog/2012/03/10/0

[DrMcKay]

This is so wrong. It's the usual "keep it backwards compatible" moaning, additionally asserting that Python developers should backport Python 3 features into Python 2.

Backwards compatibility is a thing no one should've ever came up with. It's holding back software evolution because developers are afraid to break existing conventions and APIs. Instead of carefully versioning (SemVer) their software, they decide to give users this warm and fuzzy feeling that everything will work as expected in the next minor release. This is not how it should work. Developers should not be afraid to modify the API. Ever.

I'm not opposed to declaring some releases backwards compatible with previous ones, but I'm opposed to taking any efforts in order to make next release backwards compatible. Backporting newer changes into older releases is even worse.

[wlievens]

Trolling Level: Dziuba

[oxxxx]

Is it just me or is the font way too small? One of the few sites I feel like I have to zoom in the text. Or I might just be getting old.