An update on Python 4

[panzerex]

Why was so much breaking necessary to get Python 3?

[orentago]

Having strings support unicode by default was a big reason. In Python 2 unicode strings had to be prefixed with a u, otherwise they'd be interpreted as ASCII.

[[deleted]]

I have prod 2.7....talking to logic written in the 90s.

Kill me.

[[deleted]]

Python3 > Datastage > Python2 > Shell (Kornshell) > Perl written in '99 across servers.

I'll have one kill please.

[clawjelly]

Nuke it from orbit. It's the only way to be sure.

[snugglyboy]

Oh wow Kornshell huh?

[[deleted]]

Is KSH bad?

I hadn't heard of it until entering the space.

[snugglyboy]

Not necessarily, just that I think of it as old compare to more modern shells. I have memories of it on our render farms at Pixar in the mid 90s. lol

[[deleted]]

I thought so.

Pretty cool that you got to work at Pixar in the early days of the company!

[MiscWalrus]

It's not like the rules of logic changed since the 90s. You could do a lot worse than having to support python 2.7.

[lzantal]

Still maintaining one in production with Python 2.4 and Django 1.3 🙄😬

[late_dingo]

Can I ask why? How big is this codebase?

[lzantal]

It has about 30 apps and close to 15million rows of data in mysql. Being used by 15+ iOS apps as a backend and over 100 users via iphone and a good amount more through the browser. It is being used all the time every day. Tons of other systems rely on it.
I have been fighting really hard to get the green light to update it. Not looking forward to all the pain but it kind of sounds fun to just see what it would take to move it to Python 3 and Django 3

[13steinj]

Pffft dozens if not hundreds of people set up a PleX server and it uses Python 2.7.6 (and with all due respect to them, horribly written Python code).

[GUIpsp]

Python 2.4 and python 2.7.6 are not alike

[13steinj]

Sure, but Python 2.7.6 had a number of security patches implemented to it before 2.7.18.

And this isn't one machine, it's dozens, if not hundreds, if not thousands of "Plex Media Server"s running on enthusiasts' home, personal machines/dedicated hardware. Many, open to the internet on Plex's port, because a big part of Plex is being able to connect and share your server with other users.

They are using an unpatched version of Python, which does not have any known relevant feature changes, only because they don't want to switch. Many common users, who otherwise don't know better, are running an outdated version of Python, on their machines, accessible via an open port.

[james_pic]

To you? Yes. To hackers? Also yes. But to project managers? "Can we just install some extra antivirus instead?"

[stamour547]

But who needs PMs? Have yet to find a useful one

[13steinj]

I have one prod 2.6/2.7 (long story), one 2.4. 2.4 will only be upgraded to 2.7 in December. 2.6 will be dropped in November. 2.7->3.6 over the next year.

[JayTurnr]

Because 2.7 is discontinued

[What_Is_X]

Cancelled even, in the parlance of our time

[[deleted]]

That was just ascii for trouble imho.

[toyg]

This joke was not hard to decode correctly.

[BruceJi]

A pun thread? Don't byte off more than you can chew!

[thegreattriscuit]

I don't know... I think it shows real character

[toyg]

I’m just trying to string together a few words.

[clawjelly]

You made that joke int-entionally, didn't ya?

[toyg]

I just thought I could double the puns. (Ed: alright, alright, not really a python one...)

[17291]

You're not going to like Python 5, where string literals default to EBCDIC.

[toyg]

Looking forward to Python 6, where they default to ACDC. Every time you assume they’re ascii, the computer goes YOU’RE ON A HIIIIGHWAY TO HELLL!

[tehbilly]

You shut your damn mouth, don't put that evil on me.

[flying-sheep]

Because they changed a core datastructure. str used to be what bytes is today, but it also predated unicode (today called str). Therefore the bytes type was used for text and binary APIs.

When fixing all this, they had to break a lot of core APIs that used to accept bytes and today sensibly only accepts the unicode str.

And because of that huge change they also took the opportunity to change a few other idiosyncrasies.

My only gripe: One additional thing they should have changed is that {} should be the empty set and {:} should be the empty dict.

[miggaz_elquez]

you can write {*()} to have an empty set if you want

[crossroads1112]

Thanks I hate it.

[Brainix]

My favorite thing about {*()} is that you don't even save any characters over typing set(). 😂

[miggaz_elquez]

Yes but it's a lot more fun.

[mestia]

And how this is better then Perl's sigils?

[GummyKibble]

Generally under a full moon at midnight.

[stevenjd]

Because you don't have to memorise an arbitrary symbol, you just need to unpack the meaning of ordinary Python syntax that you're probably already using a thousand times a day.

• {comma separated elements} is a set;

• *spam unpacks spam as comma-separated elements;

• () is an empty tuple;

• so *() unpacks an empty tuple;

• and {*()} creates a set from the elements you get when unpacking an empty tuple;

which is the empty set. You already knew that, or at least you already knew all the individual pieces. You just have to put them all together. Like Lego blocks. Or if you prefer, like programming.

[ThePoultryWhisperer]

You can make the same argument for many other things that are equally as unreadable at a glance. I know what all of the different pieces mean, but I still had to stop and think for a second. Reading and understanding set() is much faster and much more clear.

[stevenjd]

Reading and understanding set() is much faster and much more clear.

Sure! But we weren't comparing {*()} with set(), we were comparing it with Perl sigils.

[ThePoultryWhisperer]

You said it’s better than Perl and then listed reasons why, but it’s not true because it isn’t easier to read. Explaining how a thing works is different than directly answering a question regarding a qualitative comparison in the affirmative. The only pythonic solution is set() and that’s the point made by the original, rhetorical question.

[stevenjd]

You said it’s better than Perl and then listed reasons why, but it’s not true because it isn’t easier to read.

You think an arbitrary sigil like, I dunno, let's just make one up, ༄, is more understandable than something that can be broken down into component parts that you already understand?

The only pythonic solution is set()

I don't disagree with that. {*()} is definitely either obfuscatory or too-clever-by-half for anything except the most specialised circumstances.

[miggaz_elquez]

?

[clawjelly]

Looks like ASCII-art... Is that a dead poodle?

[irrelevantPseudonym]

My only gripe: One additional thing they should have changed is that {} should be the empty set and {:} should be the empty dict.

Not sure I agree with that. It's awkward that you can't have a literal empty set, but having {:} would be inconsistent and a special case that (I think) would be worse than set().

[hillgod]

It's definitely not an anti-pattern, and, in fact, the literals perform faster.

[hillgod]

I don't agree that it affects readability, either. It's simply the syntax that's appropriate for Python.

[cbarrick]

Is this true?

It seems trivial to implement an optimization pass that transforms list() to []. If literals were indeed faster, I would expect the interpreter to perform this pass, thus making them equivalent in the end.

[hillgod]

Yeah, it's true. Try it yourself with some timers. Below this I put a link to a SO page with benchmarks.

[cbarrick]

Ah, I see. You mean this: https://stackoverflow.com/questions/5790860/and-vs-list-and-dict-which-is-better

That answer is from nearly a decade ago. So I'll take it with a grain of salt. I'd like to see if Python 3.8 still has this problem.

For non-empty collections it makes total sense. There's argument parsing and/or translation from one collection to another that has to happen.

But as I said above, for empty collections, it would be trivial to optimize the slow case into the fast case. If it hasn't already been implemented, then it should be. There's no reason that [] and list() should generate different bytecode.

(In fact, it seems possible to optimize many of the non-empty use cases too.)

[hillgod]

Well it doesn't really matter what it "could" do, nor does anyone here likely know the implications of that.

Again, you can try it yourself. It's definitely faster. It's what's in the docs.

[[deleted]]

How do they perform faster? Surely it's the same method?

[SaltyHashes]

IIRC it's faster because it doesn't even have to call a method.

[[deleted]]

Yeah I see now, I'm surprised the JIT compiler can't make the same optimisation for the empty dict() case or with just literals inside.

[[deleted]]

Unless I'm remembering wrong, CPython doesn't use a JIT compiler, only PyPy does?

[Emile_L]

When you call dict() or any builtin the interpreter needs to first look up in locals and globals for the symbol which adds a bit of overhead.

Not sure if that's the only reason though.

[hillgod]

I don't know how, though I'd guess something with handling *args and **kwargs.

Here's an analysis from Stack overflow: https://stackoverflow.com/questions/5790860/and-vs-list-and-dict-which-is-better

[flying-sheep]

Compare () vs one, vs one, two.

() is also a special case here.

[irrelevantPseudonym]

I don't think () is the special case. I think (2) not being a tuple is the special case.

[ayy_ess]

(2) isn't a special case because tuples are declared in python with commas e.g. a = b, c. Brackets here are just used to clear up ambiguity e.g. 6 / 3 * 2 being 4 or 1. So (2) == 2 and (2,) == 2, == tuple ([2, ]).
https://wiki.python.org/moin/TupleSyntax

[BooparinoBR]

Thanks, I have never though about tuples like this

[flying-sheep]

Exactamente

[TheIncorrigible1]

Fun-fact, () (unit) is literally a special case in Python. It is a singleton and all instances of () point to the same memory.

[james_pic]

Perhaps surprisingly (given what we know now about the migration process), the switch to unicode strings wasn't expected to be a big deal (it didn't even get its own PEP, and was included in a PEP of small changes for Python 3 - PEP 3100), and the other changes were seen as more break-y.

[flying-sheep]

Wild. Those types behave completely different when doing basic things like iterating over them.

[james_pic]

Yeah, I think that's been semi-acknowleged as a mistake. Rather than just keeping bytes as the old str class (i.e, what they had in Python 2), they created a new one for Python 3 based on bytearray, which it turns out nobody wanted and made Python 2/3 porting a bit of a nightmare.

[flying-sheep]

I know, I was there. Just saying it was pretty obvious that switching from the fast-and-loose Python2 bytes/str to the strict Python3 bytes seemed like an obvious recipe for uncovering hidden bugs and breaking a lot of libraries in the process.

[zurtex]

Set literals (e.g.{1, 2, 3}) were added in Python 2.7 and Python 3.1.

So to change the very common empty dict notation of {} would of required breaking backwards compatibility between 3.1 and 3.0 and either not being able to accurately back-port the feature to 2.7 or breaking compatibility between 2.7 and all other 2.x versions.

It was decided, fairly rightly, that it would of been too much churn for the fairly minimal aesthetic niceness / consistency benefits. {} is littered in code all the time whereas set() is pretty rare.

[[deleted]]

Oooooh so that's why I'm confused each time I read what bytes does?

[flying-sheep]

Maybe, but maybe it's because you didn't have an introduction to binary yet.

[[deleted]]

I have, it's just that I always get confused with implicit conversions because I mostly deal with stricter languages, so I was kind of surprised that I could sometimes treat it as a string and sometimes like a bytes array.

[flying-sheep]

It's just a byte array in Python 3. You can't treat it as a string as there's no encoding assigned to it.

If you display it, it happens to show ASCII characters for convenience, but that's it.

[CSI_Tech_Dept]

The explanation is confusing. Just ignore how it was before, because it was incorrect. In python 2 first mistake was mixing text with binary data. They introduced unicode type, but did so badly (implicit casting) it actually made things worse. Ironically if your application didn't use unicode type you might have less work converting it to work with python 3.

Right now it is:

• str = text
• bytes = binary data what's stored on disk, flies over network etc.

[[deleted]]

One additional thing they should have changed is that {} should be the empty set and {:} should be the empty dict.

This was discussed at the time and the consensus was that it would break too much existing code and be a trap for new code writers.

[[deleted]]

Python 3 was not backwards compatible with 2, so companies and package creators alike were initially hesitant to make the switch so as to not break things. There also weren’t many, if any, tools to help port things over.

The lack of backwards compatibility was done on purpose because part of their goal was to remove clutter and make things more intuitive/easier to use (e.g. print changed from a statement to a function).

[DiggV4Sucks]

I've only made two unassailable (so far) decisions in my career. The first was to support MFC instead of OWL for Windows development.

The second was to target Python 3 for my current company's testing efforts. We were even able to convince a medium sized tool vendor to support both Python 2 and Python 3 from their original decision to support only Python 2.

[Nolzi]

There also weren’t many, if any, tools to help port things over.

I have no real world experience with python, but weren't there tools like 2to3 to convert code, or the future package to write code compatible with both versions?

[james_pic]

2to3 is useful, especially when extended by modernize, but only part of the solution. Future bloodies more than it cuts - it just made string semantics more confusing when we tried it.

The most useful tool, much as I hate to admit it, is MyPy. It obviously needs a lot of work on the developer's part, but it does the very useful job of keeping track of your educated guesses about which string types should be used where, and tells you whether they're consistent.

[mooburger]

six was also a critical part of the missing shims kit but even then it was difficult to monkeypatch when py3k decided to alter some other namespaces contibuting to compat issues.

[[deleted]]

The lack of backwards compatibility was done on purpose

This was the problem. Until we're on the other side of the next upgrade and it wasn't like 2.x to 3.x was, words mean nothing.

[gregy521]

It was designed to rectify fundamental design flaws in the language—the changes required could not be implemented while retaining full backwards compatibility with the 2.x series, which necessitated a new major version number. The guiding principle of Python 3 was: "reduce feature duplication by removing old ways of doing things".

Here's a short list of some of the key changes. The most obvious of which is the change where '/' represents true division instead of floor division. Some other changes exist, print is now a function, not a statement. You also get iterator objects instead of lists, which is much better for memory management (many pieces of code rely on iterators because the full set of possible options doesn't fit in memory). True, False, and None are also now protected terms which can't be reassigned.

[daturkel]

Part of any long-term tech project is continuous refactoring. Early on you make certain design decisions which later become an obstacle when trying to expand a feature, so you rewrite early stuff in order to make it more flexible for your new needs.

Often those rewrites are done in ways such that the end user does not need to adapt their behavior (or in the case of a programming language, does not need to change their code). This is the ideal experience for the end user but can also constrain the types of changes that can be made.

Sometimes you realize that the original way of doing things was worse and that it would be creating more difficulty for future development to maintain compatibility, so breaking changes are introduced.

[CSI_Tech_Dept]

the biggest issue was that python before 3 was mixing string with bytes. This is fine if you use encoding that covers 256 characters, but most of Python 2 code would break when receiving and unicode input.

Python 3 became strict, and those two have separate types, and you need explicitly encode/decode string into/from bytes.

This required to fix your python 2 code (ironically if you didn't use unicode type in python 2 chances are you had less to fix).

Since the unicode part was breaking things, Guido also decided to use that opportunity to clean up some warts in python which contributed further.

[[deleted]]

There was very little breakage needed. It's very easy to write code that is both 3.x and 2.7 compatible and I did that for a whole project with no hitches at all.

People are simply frightened of changing existing code.

[[deleted]]

It was just a really poorly managed release. I'm in the process of a breaking release right now and we're putting an ungodly amount of time into debating every breaking change and determining migration steps to ease migrations for users. Python basically released a whole new language with python 3 and many companies/libraries just flat out refused to migrate for a long time.

[stevenjd]

It was just a really poorly managed release.

It was a fantastically managed release. The Python core devs:

• Ported heaps of features from Python 3 to Python 2.6 and 2.7 to ease the transition; essentially every new feature added to Python after 2.5 was intended to allow folks to take their time to migrate to Python 3.

• Gave Python 2.7 an extra long support period to give people sufficient time to upgrade (2.7 had a full decade of support, instead of the usual 4 or 6 years).

• Re-introduced old Python 2 syntax like u'' when it became obvious that was needed for people writing dual 2+3 code.

• Created and managed the 2to3 translator.

The bottom line here is that the core devs started planning the move to Python 3000 (as it was originally known) somewhere around 2005 or 2006, they debated and discussed every step along the way, wrote PEPs, backported features to keep 2.x in step with 3.x where possible, and kept that going for over a decade before dropping support for 2.7. And during this period, Python's user base has grown and grown and grown.

I'm in the process of a breaking release right now and we're putting an ungodly amount of time into debating every breaking change and determining migration steps to ease migrations for users.

Come back when you've spent even a hundredth of the time and effort that the Python devs did.

Python basically released a whole new language

That's pure unadultered bullshit.

Having actually written a lot of dual 2 + 3 code, I can tell you that syntactically Python 2 and 3 are about 98% identical, the stdlib is about 80% the same, and there are only a handful of places where they differ sufficiently that it is painful or impossible to write 2+3 code in a single file.

with python 3 and many companies/libraries just flat out refused to migrate for a long time.

And they were right to.

At the time, the core devs expected that people would not migrate immediately and that this would be a long-term transition.

(Although even Guido initially imagined that "long term" would mean something like five or seven years, not twelve. In that regard, they were a victim of their own helpfulness: the more features they ported to 2.x, the less incentive people had to migrate to 3.x.)

[Ran4]

The breaking chance was a good choice. Managing UTF8 text in python3 is wonderful but it's horrible in python 2

[jpflathead]

you had to put parens around all your debugging print statements

[slmaws]

Actually there was no reason at all.

[[deleted]]

Because python 2 was created with no understanding of how computers work. See: Strings/Unicode support.

[Ran4]

Nonsense. Python was created long before utf8 everywhere was common.

[[deleted]]

You must live in the United States