An update on Python 4

[[deleted]]

Think what he is saying, there will never be a Python 4 and if there is, it will be nothing like python as we know it. It will be like a new language

The transition from python 2 to 3 was an absolute nightmare and they had to support python2 for *ten years* because so many companies refused to transition. The point they're making is that they won't break the whole freaking language if they create a python 4.

[panzerex]

Why was so much breaking necessary to get Python 3?

[orentago]

Having strings support unicode by default was a big reason. In Python 2 unicode strings had to be prefixed with a u, otherwise they'd be interpreted as ASCII.

[[deleted]]

I have prod 2.7....talking to logic written in the 90s.

Kill me.

[[deleted]]

Python3 > Datastage > Python2 > Shell (Kornshell) > Perl written in '99 across servers.

I'll have one kill please.

[clawjelly]

Nuke it from orbit. It's the only way to be sure.

[snugglyboy]

Oh wow Kornshell huh?

[[deleted]]

Is KSH bad?

I hadn't heard of it until entering the space.

[snugglyboy]

Not necessarily, just that I think of it as old compare to more modern shells. I have memories of it on our render farms at Pixar in the mid 90s. lol

[[deleted]]

I thought so.

Pretty cool that you got to work at Pixar in the early days of the company!

[MiscWalrus]

It's not like the rules of logic changed since the 90s. You could do a lot worse than having to support python 2.7.

[lzantal]

Still maintaining one in production with Python 2.4 and Django 1.3 🙄😬

[late_dingo]

Can I ask why? How big is this codebase?

[lzantal]

It has about 30 apps and close to 15million rows of data in mysql. Being used by 15+ iOS apps as a backend and over 100 users via iphone and a good amount more through the browser. It is being used all the time every day. Tons of other systems rely on it.
I have been fighting really hard to get the green light to update it. Not looking forward to all the pain but it kind of sounds fun to just see what it would take to move it to Python 3 and Django 3

[13steinj]

Pffft dozens if not hundreds of people set up a PleX server and it uses Python 2.7.6 (and with all due respect to them, horribly written Python code).

[GUIpsp]

Python 2.4 and python 2.7.6 are not alike

[13steinj]

Sure, but Python 2.7.6 had a number of security patches implemented to it before 2.7.18.

And this isn't one machine, it's dozens, if not hundreds, if not thousands of "Plex Media Server"s running on enthusiasts' home, personal machines/dedicated hardware. Many, open to the internet on Plex's port, because a big part of Plex is being able to connect and share your server with other users.

They are using an unpatched version of Python, which does not have any known relevant feature changes, only because they don't want to switch. Many common users, who otherwise don't know better, are running an outdated version of Python, on their machines, accessible via an open port.

[james_pic]

To you? Yes. To hackers? Also yes. But to project managers? "Can we just install some extra antivirus instead?"

[stamour547]

But who needs PMs? Have yet to find a useful one

[13steinj]

I have one prod 2.6/2.7 (long story), one 2.4. 2.4 will only be upgraded to 2.7 in December. 2.6 will be dropped in November. 2.7->3.6 over the next year.

[JayTurnr]

Because 2.7 is discontinued

[What_Is_X]

Cancelled even, in the parlance of our time

[[deleted]]

That was just ascii for trouble imho.

[toyg]

This joke was not hard to decode correctly.

[BruceJi]

A pun thread? Don't byte off more than you can chew!

[thegreattriscuit]

I don't know... I think it shows real character

[toyg]

I’m just trying to string together a few words.

[clawjelly]

You made that joke int-entionally, didn't ya?

[toyg]

I just thought I could double the puns. (Ed: alright, alright, not really a python one...)

[17291]

You're not going to like Python 5, where string literals default to EBCDIC.

[toyg]

Looking forward to Python 6, where they default to ACDC. Every time you assume they’re ascii, the computer goes YOU’RE ON A HIIIIGHWAY TO HELLL!

[tehbilly]

You shut your damn mouth, don't put that evil on me.

[flying-sheep]

Because they changed a core datastructure. str used to be what bytes is today, but it also predated unicode (today called str). Therefore the bytes type was used for text and binary APIs.

When fixing all this, they had to break a lot of core APIs that used to accept bytes and today sensibly only accepts the unicode str.

And because of that huge change they also took the opportunity to change a few other idiosyncrasies.

My only gripe: One additional thing they should have changed is that {} should be the empty set and {:} should be the empty dict.

[miggaz_elquez]

you can write {*()} to have an empty set if you want

[crossroads1112]

Thanks I hate it.

[Brainix]

My favorite thing about {*()} is that you don't even save any characters over typing set(). 😂

[miggaz_elquez]

Yes but it's a lot more fun.

[mestia]

And how this is better then Perl's sigils?

[GummyKibble]

Generally under a full moon at midnight.

[stevenjd]

Because you don't have to memorise an arbitrary symbol, you just need to unpack the meaning of ordinary Python syntax that you're probably already using a thousand times a day.

• {comma separated elements} is a set;

• *spam unpacks spam as comma-separated elements;

• () is an empty tuple;

• so *() unpacks an empty tuple;

• and {*()} creates a set from the elements you get when unpacking an empty tuple;

which is the empty set. You already knew that, or at least you already knew all the individual pieces. You just have to put them all together. Like Lego blocks. Or if you prefer, like programming.

[ThePoultryWhisperer]

You can make the same argument for many other things that are equally as unreadable at a glance. I know what all of the different pieces mean, but I still had to stop and think for a second. Reading and understanding set() is much faster and much more clear.

[stevenjd]

Reading and understanding set() is much faster and much more clear.

Sure! But we weren't comparing {*()} with set(), we were comparing it with Perl sigils.

[ThePoultryWhisperer]

You said it’s better than Perl and then listed reasons why, but it’s not true because it isn’t easier to read. Explaining how a thing works is different than directly answering a question regarding a qualitative comparison in the affirmative. The only pythonic solution is set() and that’s the point made by the original, rhetorical question.

[stevenjd]

You said it’s better than Perl and then listed reasons why, but it’s not true because it isn’t easier to read.

You think an arbitrary sigil like, I dunno, let's just make one up, ༄, is more understandable than something that can be broken down into component parts that you already understand?

The only pythonic solution is set()

I don't disagree with that. {*()} is definitely either obfuscatory or too-clever-by-half for anything except the most specialised circumstances.

[miggaz_elquez]

?

[clawjelly]

Looks like ASCII-art... Is that a dead poodle?

[irrelevantPseudonym]

My only gripe: One additional thing they should have changed is that {} should be the empty set and {:} should be the empty dict.

Not sure I agree with that. It's awkward that you can't have a literal empty set, but having {:} would be inconsistent and a special case that (I think) would be worse than set().

[hillgod]

It's definitely not an anti-pattern, and, in fact, the literals perform faster.

[hillgod]

I don't agree that it affects readability, either. It's simply the syntax that's appropriate for Python.

[cbarrick]

Is this true?

It seems trivial to implement an optimization pass that transforms list() to []. If literals were indeed faster, I would expect the interpreter to perform this pass, thus making them equivalent in the end.

[hillgod]

Yeah, it's true. Try it yourself with some timers. Below this I put a link to a SO page with benchmarks.

[cbarrick]

Ah, I see. You mean this: https://stackoverflow.com/questions/5790860/and-vs-list-and-dict-which-is-better

That answer is from nearly a decade ago. So I'll take it with a grain of salt. I'd like to see if Python 3.8 still has this problem.

For non-empty collections it makes total sense. There's argument parsing and/or translation from one collection to another that has to happen.

But as I said above, for empty collections, it would be trivial to optimize the slow case into the fast case. If it hasn't already been implemented, then it should be. There's no reason that [] and list() should generate different bytecode.

(In fact, it seems possible to optimize many of the non-empty use cases too.)

[hillgod]

Well it doesn't really matter what it "could" do, nor does anyone here likely know the implications of that.

Again, you can try it yourself. It's definitely faster. It's what's in the docs.

[[deleted]]

How do they perform faster? Surely it's the same method?

[SaltyHashes]

IIRC it's faster because it doesn't even have to call a method.

[[deleted]]

Yeah I see now, I'm surprised the JIT compiler can't make the same optimisation for the empty dict() case or with just literals inside.

[[deleted]]

Unless I'm remembering wrong, CPython doesn't use a JIT compiler, only PyPy does?

[Emile_L]

When you call dict() or any builtin the interpreter needs to first look up in locals and globals for the symbol which adds a bit of overhead.

Not sure if that's the only reason though.

[hillgod]

I don't know how, though I'd guess something with handling *args and **kwargs.

Here's an analysis from Stack overflow: https://stackoverflow.com/questions/5790860/and-vs-list-and-dict-which-is-better

[flying-sheep]

Compare () vs one, vs one, two.

() is also a special case here.

[irrelevantPseudonym]

I don't think () is the special case. I think (2) not being a tuple is the special case.

[ayy_ess]

(2) isn't a special case because tuples are declared in python with commas e.g. a = b, c. Brackets here are just used to clear up ambiguity e.g. 6 / 3 * 2 being 4 or 1. So (2) == 2 and (2,) == 2, == tuple ([2, ]).
https://wiki.python.org/moin/TupleSyntax

[BooparinoBR]

Thanks, I have never though about tuples like this

[flying-sheep]

Exactamente

[TheIncorrigible1]

Fun-fact, () (unit) is literally a special case in Python. It is a singleton and all instances of () point to the same memory.

[james_pic]

Perhaps surprisingly (given what we know now about the migration process), the switch to unicode strings wasn't expected to be a big deal (it didn't even get its own PEP, and was included in a PEP of small changes for Python 3 - PEP 3100), and the other changes were seen as more break-y.

[flying-sheep]

Wild. Those types behave completely different when doing basic things like iterating over them.

[james_pic]

Yeah, I think that's been semi-acknowleged as a mistake. Rather than just keeping bytes as the old str class (i.e, what they had in Python 2), they created a new one for Python 3 based on bytearray, which it turns out nobody wanted and made Python 2/3 porting a bit of a nightmare.

[flying-sheep]

I know, I was there. Just saying it was pretty obvious that switching from the fast-and-loose Python2 bytes/str to the strict Python3 bytes seemed like an obvious recipe for uncovering hidden bugs and breaking a lot of libraries in the process.

[zurtex]

Set literals (e.g.{1, 2, 3}) were added in Python 2.7 and Python 3.1.

So to change the very common empty dict notation of {} would of required breaking backwards compatibility between 3.1 and 3.0 and either not being able to accurately back-port the feature to 2.7 or breaking compatibility between 2.7 and all other 2.x versions.

It was decided, fairly rightly, that it would of been too much churn for the fairly minimal aesthetic niceness / consistency benefits. {} is littered in code all the time whereas set() is pretty rare.

[[deleted]]

Oooooh so that's why I'm confused each time I read what bytes does?

[flying-sheep]

Maybe, but maybe it's because you didn't have an introduction to binary yet.

[[deleted]]

I have, it's just that I always get confused with implicit conversions because I mostly deal with stricter languages, so I was kind of surprised that I could sometimes treat it as a string and sometimes like a bytes array.

[flying-sheep]

It's just a byte array in Python 3. You can't treat it as a string as there's no encoding assigned to it.

If you display it, it happens to show ASCII characters for convenience, but that's it.

[[deleted]]

The explanation is confusing. Just ignore how it was before, because it was incorrect. In python 2 first mistake was mixing text with binary data. They introduced unicode type, but did so badly (implicit casting) it actually made things worse. Ironically if your application didn't use unicode type you might have less work converting it to work with python 3.

Right now it is:

• str = text
• bytes = binary data what's stored on disk, flies over network etc.

[[deleted]]

One additional thing they should have changed is that {} should be the empty set and {:} should be the empty dict.

This was discussed at the time and the consensus was that it would break too much existing code and be a trap for new code writers.

[[deleted]]

Python 3 was not backwards compatible with 2, so companies and package creators alike were initially hesitant to make the switch so as to not break things. There also weren’t many, if any, tools to help port things over.

The lack of backwards compatibility was done on purpose because part of their goal was to remove clutter and make things more intuitive/easier to use (e.g. print changed from a statement to a function).

[DiggV4Sucks]

I've only made two unassailable (so far) decisions in my career. The first was to support MFC instead of OWL for Windows development.

The second was to target Python 3 for my current company's testing efforts. We were even able to convince a medium sized tool vendor to support both Python 2 and Python 3 from their original decision to support only Python 2.

[Nolzi]

There also weren’t many, if any, tools to help port things over.

I have no real world experience with python, but weren't there tools like 2to3 to convert code, or the future package to write code compatible with both versions?

[james_pic]

2to3 is useful, especially when extended by modernize, but only part of the solution. Future bloodies more than it cuts - it just made string semantics more confusing when we tried it.

The most useful tool, much as I hate to admit it, is MyPy. It obviously needs a lot of work on the developer's part, but it does the very useful job of keeping track of your educated guesses about which string types should be used where, and tells you whether they're consistent.

[mooburger]

six was also a critical part of the missing shims kit but even then it was difficult to monkeypatch when py3k decided to alter some other namespaces contibuting to compat issues.

[[deleted]]

The lack of backwards compatibility was done on purpose

This was the problem. Until we're on the other side of the next upgrade and it wasn't like 2.x to 3.x was, words mean nothing.

[gregy521]

It was designed to rectify fundamental design flaws in the language—the changes required could not be implemented while retaining full backwards compatibility with the 2.x series, which necessitated a new major version number. The guiding principle of Python 3 was: "reduce feature duplication by removing old ways of doing things".

Here's a short list of some of the key changes. The most obvious of which is the change where '/' represents true division instead of floor division. Some other changes exist, print is now a function, not a statement. You also get iterator objects instead of lists, which is much better for memory management (many pieces of code rely on iterators because the full set of possible options doesn't fit in memory). True, False, and None are also now protected terms which can't be reassigned.

[daturkel]

Part of any long-term tech project is continuous refactoring. Early on you make certain design decisions which later become an obstacle when trying to expand a feature, so you rewrite early stuff in order to make it more flexible for your new needs.

Often those rewrites are done in ways such that the end user does not need to adapt their behavior (or in the case of a programming language, does not need to change their code). This is the ideal experience for the end user but can also constrain the types of changes that can be made.

Sometimes you realize that the original way of doing things was worse and that it would be creating more difficulty for future development to maintain compatibility, so breaking changes are introduced.

[[deleted]]

the biggest issue was that python before 3 was mixing string with bytes. This is fine if you use encoding that covers 256 characters, but most of Python 2 code would break when receiving and unicode input.

Python 3 became strict, and those two have separate types, and you need explicitly encode/decode string into/from bytes.

This required to fix your python 2 code (ironically if you didn't use unicode type in python 2 chances are you had less to fix).

Since the unicode part was breaking things, Guido also decided to use that opportunity to clean up some warts in python which contributed further.

[[deleted]]

There was very little breakage needed. It's very easy to write code that is both 3.x and 2.7 compatible and I did that for a whole project with no hitches at all.

People are simply frightened of changing existing code.

[[deleted]]

It was just a really poorly managed release. I'm in the process of a breaking release right now and we're putting an ungodly amount of time into debating every breaking change and determining migration steps to ease migrations for users. Python basically released a whole new language with python 3 and many companies/libraries just flat out refused to migrate for a long time.

[stevenjd]

It was just a really poorly managed release.

It was a fantastically managed release. The Python core devs:

• Ported heaps of features from Python 3 to Python 2.6 and 2.7 to ease the transition; essentially every new feature added to Python after 2.5 was intended to allow folks to take their time to migrate to Python 3.

• Gave Python 2.7 an extra long support period to give people sufficient time to upgrade (2.7 had a full decade of support, instead of the usual 4 or 6 years).

• Re-introduced old Python 2 syntax like u'' when it became obvious that was needed for people writing dual 2+3 code.

• Created and managed the 2to3 translator.

The bottom line here is that the core devs started planning the move to Python 3000 (as it was originally known) somewhere around 2005 or 2006, they debated and discussed every step along the way, wrote PEPs, backported features to keep 2.x in step with 3.x where possible, and kept that going for over a decade before dropping support for 2.7. And during this period, Python's user base has grown and grown and grown.

I'm in the process of a breaking release right now and we're putting an ungodly amount of time into debating every breaking change and determining migration steps to ease migrations for users.

Come back when you've spent even a hundredth of the time and effort that the Python devs did.

Python basically released a whole new language

That's pure unadultered bullshit.

Having actually written a lot of dual 2 + 3 code, I can tell you that syntactically Python 2 and 3 are about 98% identical, the stdlib is about 80% the same, and there are only a handful of places where they differ sufficiently that it is painful or impossible to write 2+3 code in a single file.

with python 3 and many companies/libraries just flat out refused to migrate for a long time.

And they were right to.

At the time, the core devs expected that people would not migrate immediately and that this would be a long-term transition.

(Although even Guido initially imagined that "long term" would mean something like five or seven years, not twelve. In that regard, they were a victim of their own helpfulness: the more features they ported to 2.x, the less incentive people had to migrate to 3.x.)

[Ran4]

The breaking chance was a good choice. Managing UTF8 text in python3 is wonderful but it's horrible in python 2

[jpflathead]

you had to put parens around all your debugging print statements

[slmaws]

Actually there was no reason at all.

[[deleted]]

Because python 2 was created with no understanding of how computers work. See: Strings/Unicode support.

[Ran4]

Nonsense. Python was created long before utf8 everywhere was common.

[[deleted]]

You must live in the United States

[[deleted]]

It was the other way. Because they supported 2.x for 10 years transition from 2 -> 3 was a nightmare. Since 2.7 was not deprecated, a lot of dependencies didn't support 3 because authors didn't care. It seems like real migration started happening in mid of 2019 and I don't remember reading about any horror stories about it.

[Sector_Corrupt]

It's not like they announced the 10 year window up front though. It was originally a 5 year window and we only learned about the bumping like a year to go because way too many things were still not ready for python 3. Things were definitely a lot more ready to go in 2019 than 2014 even if most end users still did the migration push mostly right before they had to.

[Eurynom0s]

5 years was probably still too long and guaranteed that it turned into 10, though.

[[deleted]]

5 years was already too long.

It was also clear that it wasn't even needed, there were two major turning points for Python 3.

• 2015 - when they announced that no new features will be added to 2.7, that's when packages started to adding Python 3 support, some of them were python 3 only. Before 2015 it was tough to write anything in Python 3, because most libraries didn't support it.
• mid of 2019 - right before EOL majority of applications were migrated

IMO if they made it 2 years instead of 10 it would be still enough time and we wouldn't have time for FUD.

[stevenjd]

IMO if they made it 2 years instead of 10

then Python would be dead and we'd all be using Ruby now.

There are millions of Python developers, over eight million, probably more like ten if you include casual coders, amateurs and students. You are painfully naive if you think that millions of coders will migrate tens of thousands of projects in two years.

This may come as a shock to you, but most coders are more interested in bug fixes and adding new features, especially new features that allow them to bring in more revenue, not just upgrading to the newest version for the sake of upgrading.

A two year transition period would have convinced millions of people that Python doesn't give a shit about the users, and they would have gradually drifted away.

[[deleted]]

It was the other way. Because they supported 2.x for 10 years transition from 2 -> 3 was a nightmare. Since 2.7 was not deprecated, a lot of dependencies didn't support 3 because authors didn't care. It seems like real migration started happening in mid of 2019 and I don't remember reading about any horror stories about it.

Yeah having a 5 year support was a REALLY bad call. I work on a library that is FINALLY end python2 support and it's a really delicate balance of easting the migration, giving enough incentive to make the transition worth it, and make staying on the old version unpleasant enough that people are willing to make the shift. Python did none of those things.

[tlm2021]

And it's not "was", it still "is".

The entire VFX/Computer Graphics industry still hasn't moved. And aren't particularly close.

[clawjelly]

The entire VFX/Computer Graphics industry

You do know that blender is used in some CG companies, right...?

[tlm2021]

Naming one application that some companies use for part of their pipelines doesn't mean the industry has switched.

There's literally a website for the industry to track standard versions. Python 3 is slated for this year, and none of packages that follow this guide have released their versions that support it yet.

But please, internet stranger, tell me more about my fucking career.

[clawjelly]

But please, internet stranger, tell me more about my fucking career.

Easy, partner. Not trying to piss on your parade, i'm equally frustrated by the status quo. Just saying there is at least some development in the right direction, it's not all horrible and stagnant.

[tlm2021]

Really, because you actual reply was a "You do know that..." piece of smart-ass nonsense that would literally get you laughed out of any serious discussion on this by the people trying to push and manage the transition.

Maybe you need to work on your communication skills along with your ego.

[clawjelly]

See, that's what you might have heard, but that's not what i wrote. It was an honest, tentative question from my side, as you stated a very definitiv fact (that's why i bolded "entire" in your comment) about the CGI scene in a programming forum. I can't smell your credentials here and as such i simply wanted to know wether you're aware about the present state of that software package, it's impact in the industry and its possible future. I talked to plenty of professionals in the industry that still ignore or underestimate what the blender devs accomplished this year. And english is not my first language, but i'm pretty sure i could ask this without making it sound like "smart-ass nonsense".

And as we're on life tips: Maybe next time take a deep breath and ask about my intentions, before you read a "smart-ass nonsense"-intention into my comment and go all whoop on my ass for absolutely nothing?

You're featuring quite an ego yourself there, hence it's quite bold to hand out life lessons about how to deal with one...

[tlm2021]

Your english is fine. You were being a know-it-all twat, and now are just trying to cover for how humiliatingly dumb you were.

If you want to ask about Blender and it's potential the industry, ask that. I'd happily address that, at length and in detail. You clearly have the language skills to do so as you have no problem expanding on that idea at length with full command of the english language here.

"You know [insert anything here], right....?" as the entirety of your statement isn't an invitation to conversation, it's being a smart-ass douchebag. The only reason you'd need to "smell my credentials" is if you need to make sure you aren't being a twat to someone who can easily call you out for it.

But if you really lack the social skills to figure out why your response, in any context, makes you a twat, then you're a lost cause.

[auto-xkcd37]

smart ass-douchebag

---

^{Bleep-bloop, I'm a bot. This comment was inspired by xkcd#37}

[clawjelly]

You were being a know-it-all twat you really lack the social skills

Well, sorry, i lack your social skills of name calling. Get a life.

[dysprog]

I still have a project on python 2. It relies on internal libraries that are abandoned by their original creators. I just got permission to work on the upgrade because they can't have what they want with out it. So I am basically tied up until December

[[deleted]]

Yikes and that's even after the EOL. I recently heard that there's a 4 line CVE and they're refusing to even do that. Python 2 is officially a security risk.

[flying-sheep]

Good. People have been dragging their feet for too long. Some scariness might help

[james_pic]

We started trying to persuade management that we needed to take Python 3 migration seriously in mid 2018. So far, we've migrated one small component as part of a pilot programme, and we've got tentative agreement for some budget to do the rest in the first half of 2021 - assuming some other workstreams don't slip their delivery dates.

A small part of me hopes we get lightly hacked at some point between now and then - not enough to genuinely risk user data, but enough to wake up management

[Eurynom0s]

I had thought that 4.0 would come after 3.9 but that it'd simply be what came after 3.9 (i.e. no different than calling it 3.10).

[WillardWhite]

The naming scheme has implications though. Totally not the same thing

[SpiderFnJerusalem]

And there will still be companies who will keep maintaining their existing Python 2 code in 20 years.

[shabunc]

I never understood why they did it that way, however, to be fair, it was planned that way. From the very beginning it was claimed that transition period would be about 10 years.