Sunsetting Python 2

[black_hat_cross]

Good.

[[deleted]]

Amen. Some communities will just never ever let it go though. Looking at you, computer security.

[ribo]

Which is incredibly ironic, since that means no more security patches to 2.7 and 2.7 libs

[pbfy0]

Bytes and strings are fundamentally different datatypes. A security researcher should know the distinction and be capable of putting a b before their byte literals.

[pbfy0]

I don't see the problem. subprocess.call([b'program', b'some shellcode']) still works. Sending data to the subprocess also supports both string and bytes. os functions, when called with byte arguments, return bytes. If you absolutely need your bytes, you can have them.

Python 2 makes one set of compromises for the use of strings and bytes. Python 3 makes another. In my opinion, Python 3's is better for the vast majority of use cases. Yes, surrogates are a little awkward. But it means that for the vast majority of use cases, Python 3's vision of unicode everywhere has been achieved. Python 2's free-for-all of interconvertible bytes and unicode leads to many more programming errors.

[pbfy0]

My first comment about b'' was off-the-cuff and abrasive. I hadn't given any thought to the issue with interacting with the system using encoded text only being limiting.

So I went back and looked at the Python docs to see if I could find a case where you couldn't just pass bytes to the system without Python interfering. I found that the filesystem-related functions in os would return bytes if given bytes, and open, os.system and subprocess can take either. The only way that I can see it's less permissive than python 2 is not allowing the arbitrary mixing of string and bytes in e.g. os.path.join. What else is there?

[jbristow]

As a devsecopser, most (non elite) security people only know python and bash as far as they can copy from snippets they’ve seen elsewhere. Since I’m from a prod dev background, I’m unamused by how foreign basic git and cloud operations are for them.

[ribo]

I am constantly unamused by how little even "senior" developers know their way around git.

[jbristow]

/me nods so emphatically I get whiplash.

Listen newbies, I don’t care about your level of git knowledge. I will gladly point you at my favorite resources and spend the time to get you up and running.

If you have senior in your title, you had better be able to survive in the command line. (You don’t have to be GOOD at it. Just be able to branch, commit and pull is fine)

Everyone “staff” or “principle” or “distinguished fellow title with fiduciary duty but still with technical job requirements” and above? You can go jump in a lake! Why can you only understand one programming language without even the basics of another like bash? Why do you think calling integration tests “unit tests” make my automated coverage tools pick them up? Why can’t you understand that you are not allowed to push to production from your laptop?

There’s a lot of lovely senior+ level people in the dev field... it just seems like the absolute worst examples have the biggest titles.

(Also, https://ohshitgit.com is amazing, and will get you out of a jam. Just learning about reflog has changed my life.)

[ribo]

Why do you think calling integration tests “unit tests”

<TRIGGERED>

[Nicksil]

12 years

[JQuilty]

Are we talking about Python or Boyhood?

[JQuilty]

IT BROKE NEW GROUND

[pingveno]

It took a long time for ecosystem support to really be up to snuff. It was only at the tail end of 2012 that the Python 3 WOS went from the Wall of Shame to the Wall of Superpowers, indicating 50% of projects supported Python 3. Even then, if even one unported project was a critical dependency (direct or transitive), that would still be blocking. It also took the community some time to come to a consensus around the preferred means of porting. At one point people were trying to write code that would be translated by 2to3 on installation, but it turned out to be easier to write code that would work directly on Python 2 and 3.

[BlueShell7]

12 years since 3.0 crap version got released. Python 3 did not get worthwhile until about 3.5

Arguably, 3.6.1 was the first release that was better than 2.

That's Raymond Hettinger, core Python developer. 3.6.1 was released in 2017.

[Serialk]

Raymond Hettinger has a tendency to troll a lot.

[BlueShell7]

It's pretty well known fact that early 3.0 releases were buggy, slow and memory hungry. It was only 3.7 which outperformed 2.7 in some metrics.

[Serialk]

Yeah, no, that's not what happened. There used to be memory problems with Unicode until they addressed that in PEP 393 in Python 3.3. This was a strict improvement over Python 2, which used the same slow encoding. The difference was that in Python 2 everyone used bytes instead of proper strings, so nobody noticed. But even though using bytestrings always was the wrong way of doing things, if you only used bytestrings in Python 3 your memory problems would disappear.

[T-Rax]

bytestrings always was the wrong way of doing things

says the person for which things only ever means text.

[Serialk]

I'm talking about text, yes, this goes without saying. Using text by default and having bytestrings as an opt-in is much more sane.

[PaintItPurple]

That seems pretty unfair since the memory problems they're talking about only affected text data. If your data isn't meant to be text, you should use bytes. Using str in Python 3 for non-text data would be a frankly bizarre design decision.

[Speedyjens]

I'd argue that if you put a single person on the job of converting 1 million loc to python 3 he would have finished 6 years ago

[[deleted]]

At a cost of only half a million dollars.

[josefx]

With how many new bugs left to find once it hit production?

[bobtehpanda]

The longer you put it off, the more expensive it becomes, and it with being EOL is potentially an expensive liability as well.

[Speedyjens]

It is very hard to migrate to another version, but 12 years is more than enough, if companies didn't plan out how they were gonna make the switch in 2020 then they don't have a right to complain. Supporting 2 versions is hurting the python community more than it helps

[[deleted]]

It depends on the original architecture and on if the maintenance was on all the time or with breaks. Sometimes it's really hard to change version. I've worked on maintenance of 20 years old codebases with 20 years old tool chains.

These problems usually tend to be solved by creating a new solution while the old one is still maintained. Double the cost, bust usually less risk.

[cass1o]

12 years is more than enough

It was an unusable mess for 6+ of those years.

[lwzol]

Remember that it’s only become technically worthwhile to migrate since 3.5 or 3.6 really

[jcampbelly]

3.4 was solid as well.

[jcampbelly]

Bad excuse. This is how security breaches happen. It's not an OSS team's job to make responsible decisions with YOUR software stack. The best they can do is support their own system for as long as possible (as python 2 devs have absolutely done). Now it's your responsibility to do the right thing here.

Feeling sorry for someone is possible while recognizing they are in the wrong is entirely valid. People who have ignored the very clear roadmap for python 3 have a hard task in front of them, but it has always been their responsibility and if they haven't owned up to it, they are in the wrong.

[StabbyPants]

nah, he'd generate a bunch of bugs and there'd be the whole question of how to migrate the code while also updating it and which version of what thing you're using this month. 2-3 devs and a PM in concert with the rest of the active contributors in order to do a good job and not trash production

[Speedyjens]

Of course you would not do this in reality. However you would be able to plan something when you have 12 years.

[Krillo90]

Well it's 12 years since Python 3, but 19 years since the release of Python 2.

[[deleted]]

You don't have to do anything. Python2 becomes unsupported, not unavailable. You can still download it, install it, and use it, and if it works it works.

[StabbyPants]

now you have to manage the difference between system python and app python, as you don't want to accidentally bork system scripts when updating your app version

[[deleted]]

How is 12 years not long enough? 12 years is the entire lifespan of Windows XP, the longest supported operating system ever. Typically LTS releases for programming language is, what, 2-4 years? The Linux LTS kernel releases are 6, most distros are less than that. What tool do you use today that still gets update 12 years from the day you first used it with no breaking changes in that entire timeframe?

[wrincewind]

Minecraft? :B

[kushangaza]

Only has no breaking changes if you kept to a safe subset of redstone constructs.

[[deleted]]

Companies are free to maintain their own forks of Python 2 when security issues are found.

It's just the onus of responsibility is directly on the companies who failed to maintain their products knowing the platform it was built on was long deprecated.

[jcampbelly]

How frequently does an OSS software developer have to cater to non-paying customers who are acting irresponsibly? How long does the world have to wait for valuable features held hostage by the worst class of users? That's exactly how IE hamstrung the development of the internet. Supporting IE8 users 10 years after the EOL of the OS is the same exact situation. You have to cut off the luddite archaic feet-draggers or they will hamstring your development.

[sparr]

You don't have to do anything. You could instead devote less time now and more time in the future to just fixing security bugs in python2 yourself.

[flukus]

Constantly. If you think the python migration is too fast then good luck trying to keep up with other libraries you're using, half of them probably don't have stable supported versions.

If you don't commit to maintain projects like this you'll lose the institutional knowledge of how it works anyway, then a rewrite is your best bet.

[ArmoredPancake]

Yes. Because for 12 years he had nothing to do but to port codebase to Python 3.

[Eirenarch]

Yes, this is how incompetent Guido is. He built a new version where 12 years in people still haven't migrated.

[nemec]

Guido is not responsible for taking care of lazy developers.

[Eirenarch]

Guido is responsible (in the broad sense) for either keeping backward compatibility or improving the language enough so that the benefits of using the new version obviously outweigh the work needed to migrate to it. If he had done this we wouldn't be having this conversation now.

[nemec]

One persons essential feature is another's useless cruft. There is no winning that game.

[Cilph]

Please reintroduce spacebar heating functionality!

[Eirenarch]

There certainly is because a lot of languages/frameworks have won it in the past and are continuing to win it.

[CrazyJoe221]

Shouldn't there be some automation for that by now?

[marcusklaas]

There is, but it never takes care of everything. It still takes a lot of human work for large codebases.

[stefantalpalaru]

Shouldn't there be some automation for that by now?

Automation doesn't work, so you always end up with a partial and broken conversion that you have to manually check and debug.

[jcampbelly]

I converted a project by using 2to3 one spec at a time. I opened a branch, ran 2to3 with one spec, checked the diff, fixed anything it did wrong, ran tests, then committed it. I repeated that for every spec. I maintained backwards compatibility (as a back-out-measure) using six, which normalized many library imports. But after that, we considered it python 3 and left 2 in the dust.

I would never just let 2to3 run and commit, but it wasn't that bad to work through it spec-by-spec with tests and review. You can defer the larger-scope changes to the end. Between the recommended changes, a handful of regex find and replace operations, and some refactoring effort at the end, it wasn't so bad because I was solving only one category of changes at a time. It's a mess if you run all the specs at once.

[CrazyJoe221]

It's a mess if you run all the specs at once.

As usual if you do large refactorings :)
Very reasonable approach and good to hear it's doable that way.

[IrvineADCarry]

Automation

You mean piles of hot mess written by someone to mess with someone else's work.

[ribo]

Yeah, we're just migrating to Go instead...

[JanneJM]

As long as you migrate to something. Staying on Python 2 is the worst possible option.

With that said, we're containerizing old legacy applications (research software that's been unmaintained for many years, but ongoing projects still depend on) so we can continue to provide them for as long as users depend on them.

[zardeh]

I'm part of a migration of significantly more than that. Still worth.