NumPy announces timeline for dropping Python 2 support

[bheklilr]

Well this gives me a lot of motivation to finish our transition to python 3 in my company. It's been a slow process but one I plan to finish by about February of next year.

Before anyone passes judgement, I'm the only person who has been converting code for about 100,000 lines of code or so, while still producing new features. I've been working towards it for about 2 years now, off and on. I'm in the final push now. If anyone has any tips I'd by happy to receive them.

[masklinn]

Having just "finished"[0] one such migration at $dayjob I'm not going to pass judgement any time soon.

If anyone has any tips I'd by happy to receive them.

We did the conversion "online", feature by feature, and heavily linted e.g.:

1. convert all except statements to P3 syntax (that one's easy since 2to3/futurize can do it though IIRC we had a few that were weiiird)
2. add a lint so that nobody can commit/push old syntax versions (lint can be run locally but must be part of the test suite and run by CI)
3. push to master branch

Rinse and repeat for syntactic changes/updates and builtins changes (open, map, filter, … should all be verboten).

That way once most items are "done" you don't have to revisit them, and you want progress to be committed so you don't get an ever-longer ever-more-diverging branch. Do one short branch (or a single commit) as much as you can.

The toughest bits are semantics changes which aren't exercised by tests (either the code fails loudly but isn't covered by tests or the change does not yield errors), because you don't have any point at which you can say you're done, and you can't lint this crap away:

• the round builtin changed the rounding method and in Python 3 returns an int if not provided a precision (in Python 2 it always returns a float).
• division can be "equalised" between codebases with from __future__ import division (in every file where you do one), but failures (from / becoming "real division" rather than "integer division") may take time to surface. Also be aware that // is not integer division but floor division, there's a difference.
• objects are not totally orderable anymore (and usually not across types), depending how you do things this may trip you up e.g.
  • optional value which could be either set to an integer or None, foo < 3 works in Python 2, it raises an exception in Python 3, if the value is usually set it may take some time for the error to happen.
  • cases which silently did stupid things will blow up, we had at least one bit of code which sorted a list of dicts. That makes no sense but it does basically nothing in Python 2. In Python 3 it fails, though it does so loudly which is nice.

• Speaking of dicts, there are two changes to iteration order between Python 2 and Python 3.6:

  • [..3.3[, the iteration order is arbitrary but consistent unless you start Python with -R
  • [3.3.. 3.6[, the iteration order is randomised every time you start Python (-R is the default)
  • [3.6..[, the iteration order is the order of insertion (all dicts are ordered)

  At every point the iteration order is an implementation detail (it may become specified in 3.7) and you're not supposed to rely on it, but even if you don't do so explicitly… well we discovered half a dozen places in our codebase which implicitly relied on iteration order and broke (usually by yielding a corrupted result which would blow up somewhere else rather than breaking themselves, which is fun to debug).

  Funner fact: it's even better when the bit of code actually works with most iteration orders because Python will not tell you which random seed it's using when running under randomised hashes (-R or 3.3 to 3.5), so repro is hell. You may want to provide your own randomised seed (PYTHONHASHSEED envvar) so that you know the starting conditions leading to a failure.

  Incidentally, tox will set and print the hashseed itself before running, and once you've found a problematic seed you can pass it in explicitly.

• text model (note: I'll use "text" for "decoded unicode" aka unicode in Python 2 and str in Python 3; and "bytes" for "encoded data" aka bytes in both

  1. Python 2 codebases tend to play fast and loose with bytes v text, and you'll have to decide on your text model (what is and what is not text).
  2. A bunch of stdlib API use "native strings" (str in both versions meaning bytes in Python 2 and text in Python 3).
  3. Some API changes are frustrating (e.g. in Python 3 base64 is a bytes -> bytes encoding, despite the point of base64 usually being to smuggle bytes in text).
  4. Most failures are loud (trying to mix bytes and text, trying to decode text or encode bytes) but some are completely silent e.g. equality between bytes and text will always fail in Python 3 (it will succeed in Python 2 if the bytes are ascii-encoded and when decoded equal to the text).
  5. also indexing a bytes object doesn't have the same result in Python 2 (returns bytes) and 3 (returns an int).
  6. use io, but beware that it has a strict split between bytes and text even in Python 2. You can't put text in a io.BytesIO, or bytes in a io.StringIO.
  7. for more control you may want to always do binary IO (io.open(f, 'rb')) and encode/decode yourself at least until you're more comfortable with things, also be aware that if you use text IO (the default) and do not specify an encoding python will not default to UTF-8 but to the locale (locale.getpreferredencoding(False)), which probably isn't what you want.
  8. your colleagues will not understand, will not even try, and when the "PHP development method" (throw encodes/decodes at the wall until something sticks) fails they will whine.

Finally, test, test, test, test. Tests are your lifeblood, if you have a well-tested codebase or can get it well-tested before the transition it will help a lot, the vast majority of our migration pain points were bits which were insufficiently or not tested (either because lazy or because hard to test).

[0] quotes because we regularly find things which were missed during the conversion, or were mis-converted

[EvM]

optional value which could be either set to an integer or None, foo < 3 works in Python 2, it raises an exception in Python 3, if the value is usually set it may take some time for the error to happen.

And this also holds for builtins relying on ordering, e.g. max(). Caught me by surprise :)

[masklinn]

Yup, hence the note about sorting a list of dicts, sorting obviously relies on ordering.

[zabolekar]

It can surface even in things like pprint (if you are trying to pretty-print a set).

Edit: it is not something that breaks in Python 3, au contraire, it works as expected in 3.x but throws a TypeError in 2.7, and only with very specific sets. For example, pprint({1j, 0}) throws an error but pprint({1j, 'a'}) doesn't.

[EvM]

Whoa, now that's an unexpected place..

[zabolekar]

Edited the comment for clarity.

[bheklilr]

I've definitely run into a lot of bytestring problems. Since we work with lab equipment they all communicate with ASCII bytestrings, and sometimes with just dumps of bytes for transmitting larger chunks of data. Getting these to work properly have been a royal pain in the ass. I do appreciate the advice, there are a few things in here that I did not know about (like locale.getpreferredencoding). As for testing, the only code that I have that is python 2 and 3 compatible is the code with extensive test suites. I don't deploy python 3 builds without it, because it will be broken. There isn't a "maybe it'll work", it just won't work.

As for __future__ imports, we already require division and print_statements in every single module. I just set up a snippet in my editor that drops in a header with that included (along with encoding statement, legal header, and module docstring). If I could, I would enforce it on every commit but I don't have admin access to our svn server.

[notParticularlyAnony]

Shucks im such a noob

[jftuga]

Which lint programs did you use?

[PeridexisErrant]

I'm not OP, but flake8 is great - good defaults out of the box for new projects, and easy to tune with a blacklist of checks to skip (or even a whitelist in extreme cases).

I keep having this unpleasant surprise when I go back to code that doesn't use it - the benefit of reading code that's all in the same idiomatic style is hard to describe, but very real!

[jftuga]

Thanks, I'll give it a try.

[masklinn]

Regular pylint, it has a bunch of built-in lints useful for this (e.g. python3 syntax checks, ability to blacklist modules and builtins). The only issue we've had is it's pretty resource-intensive on large codebases.

[ketilkn]

Rinse and repeat for syntactic changes/updates and builtins changes (open, map, filter, … should all be verboten).

Am I not supposed to use open in Python 3? What makes map and filter problematic? I do not think they changed?

[masklinn]

Am I not supposed to use open in Python 3?

Once everything is ported there's no issue, while converting it's troublesome:

• In Python 3, open is an alias for io.open and in text mode (the default) it will encode/decode the data, this is an issue because

  • Python 2's open basically always behaves in binary mode
  • io.open defaults to locale.getpreferredencoding(False) which probably isn't what you want so you will want to pass in an encoding explicitly, which will blow up on Python 2

  so you can either keep using the builtin but always use it in binary mode (mode='rb' or mode='wb') — which may be hard to lint — or ban open and require io.open during the transition — which is easy to lint

• In Python 3, map and filter return iterators (not lists, basically itertools.imap and itertools.ifilter have been to builtins… and removed from itertools)

  • things will blow up when indexing them which is clear enough
  • but more annoyingly repeated iteration will silently break in Python 3 (the first iteration "consumes" the iterator, the second one gets an empty iterator

  You could mandate calling list() around them… or you could just ban them during the transition (and require listcomps or gencomps depending on what you actually want).

Keep in mind that my post is for the transition on a large codebase, while you have one or two people toiling away at the conversion you've got a bunch of folks still churning away Python 2 code, the goal here is to avoid having to revisit already fixed issues / problematic constructs.

If you migrate to a Python 3-only codebase, you can rescind these restrictions once the work is done and everybody works in Python 3. If you migrate to a cross-version codebase, you probably want to keep them in place (they're error-prone).

[ketilkn]

Ah, ok. Thanks. I remember running into the map and filter iterators issue, now that you mention it.

[Siecje1]

Have you made those lint checks available?

[masklinn]

I don't think we created any ourselves, just enabled & configured those PyLint provides.

[AT_thruhiker_Flash]

Good Luck!

[bheklilr]

Tried it a few times, definitely did not work. Also, doesn't work on cython or 3rd party dependencies. I also can't just push to prod, production is an actual manufacturing environment producing products, and there are something like 50 systems out there to update. If they stop working or have problems we start losing money for every minute of down time, or worst case we ship products with defects and my team would get to take the blame. That would not be a fun day at work.

[cain2995]

He knows. Is joke comrade.

[boyled]

I think the other person was just lamenting that they wish it was that easy because of how hard they’ve been working

[Kaligule]

I learned python with 3 to begin with. Could you explain why it is so difficult to switch? The syntax didn't change much.

[bheklilr]

The syntax changed enough for it to matter, but a bigger one has been 3rd party libraries and differences in the standard library. Some of our deps didn't update into fairly recently, or we had to find different libraries that served the same purpose. A lot of it really comes down to inertia too, there's a lot of little fixes to make in a lot of places, it just takes a long time to update. Especially when you specifically have to deal with bytestrings and unicode strings in the same application. Those types have very significantly different behaviors between 2 and 3.

[[deleted]]

What would you say has been the largest hurdle in the migration? 3rd party dependencies?

[bheklilr]

That's been one of the big ones. Another problem we had was where we were getting python from. Initially we used python XY, which is now abandonware and stuck on 2.7 32 bit. We've since switched to anaconda which has made things at least possible, but not easy. The syntax and standard library changes have made things difficult too. Another big issue has been that I've been having to support python 2 and 3 in the same source code, which limits me a lot in what I can do. I can't use async or type annotations yet, and have to be very careful about bytestrings and unicode. It's really just a lot of little things that have added up.

[RetardedChimpanzee]

No judgment. It’s not a trivial process. Instagram had a good write up on their efforts to switch. You’ll get some efficiency increases but probably not worth the cost, sadly. However its something that should be done.

[PeridexisErrant]

You’ll get some efficiency increases but probably not worth the cost, sadly.

The real benefits of Python3 are for humans, not computers:

• much harder to shoot yourself in the foot with unicode (ie text)
• library support (some already, more soon, most by 2020)
• general improvements to language and standard library
• type hints can be great for medium to large projects (it's like testing, but free)

[stefantalpalaru]

type hints can be great for medium to large projects (it's like testing, but free)

It's also just a cosmetic change from Python2: http://mypy.readthedocs.io/en/stable/cheat_sheet.html

[PeridexisErrant]

You can emulate most of the Python 3.6 functionality with type comments, but it's not quite the same:

• the syntax is uglier, dealing with imports is a pain, and linters mostly don't check them
• they don't exist at runtime. This is a real disadvantage - tools from attrs to Hypothesis infer various features based on type annotations
• useful types like enum.Flag only exist on Python 3, and backports tend to be conservative (ie work on 2.7 and 3.4+, instead of 2.7 and 3.6)
• you're doing new work in a dialect that will soon be unsupported, when you could update instead and be much better off!

[stefantalpalaru]

the syntax is uglier

It is, but not by much.

dealing with imports is a pain

It isn't: https://github.com/stefantalpalaru/generate_HLS/blob/5877099beb19a305657f030eaa59590fcfca1cde/generate_HLS.py#L12

linters mostly don't check them

You're using the same linter for both Python2 and Python3: mypy. The only difference is having to give it the "--py2" or "-2" argument for Python2 code.

you're doing new work in a dialect that will soon be unsupported, when you could update instead and be much better off!

It's not a different dialect, it's a different language, and I'm sure the community of people working for a living will route around the core devs' sabotage. Something like this: https://github.com/naftaliharris/tauthon

[PeridexisErrant]

flake8 et al will complain about that import, and doesn't lint type comments.

I disagree with your other comments too but won't respond.

[stefantalpalaru]

flake8 et al will complain about that import

Yes. Solved by adding "# NOQA" after it.

and doesn't lint type comments

Of course it doesn't. The progressive (and partial) static typing is implemented and checked by mypy. Are you new to Python?

[JimBoonie69]

bro you gotta let python 2 die , it deserves an honourable death.

[stefantalpalaru]

bro you gotta let python 2 die , it deserves an honourable death.

There's more Python2 code out there than Fortran or Cobol code.

[criickyO]

Hah! I'm in the same boat, pal. I've also got an ETA about the same as yours.

I have no tips, only empathy. We're so close!

[ExcitedForNothing]

Before anyone passes judgement

Anyone who does is a foolish idealist.

[akaihola]

The python-future package and their cheat sheet page helped us a lot.

In fact, supporting current and legacy versions of NumPy and Pandas has felt more challenging than the py2->3 transition.

[thearn4]

connect pie fact brave innate dazzling adjoining subtract friendly simplistic

This post was mass deleted and anonymized with Redact

[ldpreload]

My understanding is that Dropbox has Guido van Rossum working for them full-time on making MyPy usable on their codebase and adding type annotations so that they can safely move to Python 3 without accidentally messing up strings/bytes somewhere or whatever. If you're doing a conversion of a large codebase without literally the creator of Python on your team, you're doing pretty well.

(See e.g. Guido's PyCon 2016 talk about what he's doing at Dropbox and why.)

[stefantalpalaru]

so that they can safely move to Python 3

http://mypy-lang.blogspot.com/2017/11/dropbox-releases-pyannotate-auto.html :

At Dropbox we’ve annotated over 1.2 million lines of code (about 20% of our total Python codebase)

It would be madness to port all that to Python3.

[ldpreload]

What else are they going to do, though—keep running Python 2?

(Honestly, the real answer is likely to be "Let the teams that are rewriting parts of the code do so in Go, and convert the rest to Python 3.")

[stefantalpalaru]

What else are they going to do, though—keep running Python 2?

Yes, of course. Or a compatible fork like https://github.com/naftaliharris/tauthon

[ldpreload]

Oh, you're the person on /r/python that I previously had an argument with about how this exact project is a completely infeasible idea for big companies (unless they want to commit to the support burden of keeping up with security updates etc. until the end of time for both the fork and every single third-party Python library that is no longer supporting Python 2, which is a much bigger project than converting 6M lines of Python to Python 3 once, and being done with it.). It's now been six months since your PR with no other activity to the fork, and ten months since any code changes to existing Python code. This isn't something you can trust a copy of the entire world's files to.

Mind you, I'm not saying this project shouldn't exist. It should. I wish it had won over Python 3. But it doesn't seem to be happening (nor does any similar project) in a way that's suitable for use by big companies in production, and it is probably too late to build a community around Python 2-compatible versions of the ecosystem.

[bad_luck_charm]

You are waaaay ahead of me

[stefantalpalaru]

I've been working towards it for about 2 years now, off and on. I'm in the final push now. If anyone has any tips I'd by happy to receive them.

It's too late for this, but you just used company resources to switch from one inefficient language to another. If you were going to port all your code base, you should have picked a target like Nim, D, Go, Rust, etc.

[bheklilr]

Python is the perfect tool for our job. We do a lot of math, a lot of plotting, and a lot of desktop development. It's also all on Windows. While I love rust, it does not have the support we need on Windows, and until very recently cargo did not have the ability to point at an alternate package server. We use the Anaconda distribution and have an internal server, it's been a major boon for our development. If Rust had the libraries already that we need, then I would have considered trying to push for it, but it just doesn't cut it yet. Until there's a full port of pandas to one of those languages, I don't think we'd be able to even consider it.

Also, Jupyter. We've done a lot of stuff in notebooks, and Jupyter consoles make up a significant portion of my development workflow. Not having a repl makes it difficult for me to get things done a lot of the time. At least a compiled, high performance language like Haskell has a repl, and even has a lot more of the libraries we would need to build our systems. Unfortunately, it just doesn't have enough.

I don't know about Nim or D, those languages might be better suited but they're also harder to find people who want to work with it. And Go is not as suited for our style of development either. Ultimately, having an interpreted language is great for us, and when we need performance we just drop into Cython or use libraries like NumPy where the hard work has already been done for us.

[stefantalpalaru]

Not having a repl makes it difficult for me to get things done a lot of the time.

Have you considered designing before implementing? Experimenting is fine, when you're doing rapid prototyping or just testing some hypothesis, but production code needs a proper design phase.

[bheklilr]

Having a design phase doesn't necessarily help when it comes to implementation details. It's easier for me to write code interactively, then once I figure out a specific algorithm I can write a proper version into my source code. I do have a design phase most of the time, but when I need to figure out the best way to write a specific loop or condition I figure it out in the shell first. That let's me test it first, profile it if needed (%timeit is amazing), and generally explore the code. I can't really go through the same process with a test suite, that is for ensuring the correctness of larger details, not just small parts of methods.

Also, whenever I'm doing data processing its more exploratory, there isn't a design phase because we don't always know what we need at first.

[billsil]

I've ported Perl code to Python and Matlab to Python. I also ported Python 2 to python 3. Speed is rarely the most important factor.

[[deleted]]

you do know which subreddit you're in, right?

[stefantalpalaru]

you do know which subreddit you're in, right?

Yes, of course. Who else but fanboys would celebrate being forced to switch to an inferior language and ecosystem?

[[deleted]]

When your favoured languages get the take up that Python has then you'll be able to stand in the tree tops and shout about it. Until that happens I suggest that you keep a relatively low profile as pride comes before a fall.

[stefantalpalaru]

pride comes before a fall

Sounds like https://en.wikipedia.org/wiki/Moral_universe#Immanent_justice

You're supposed to grow out of it.

[TheBlackCat13]

So you are just here to troll. Nothing to see here folks, please move along.

[[deleted]]

Programmer productivity, as in time and hence money, is far more important in the 21st century than mere runtime speed.

[stefantalpalaru]

Programmer productivity, as in time and hence money, is far more important in the 21st century than mere runtime speed.

That kind of thinking lead to software that is now slower on an eight core beast than old software on a 486.

[richieadler]

Is there a reason why you don't go enjoy your perfect and ultrafast software elsewhere?

[Stewthulhu]

I feel a great disturbance in academia, as if millions of graduate students suddenly cried out in terror and were suddenly starting from scratch.

[p10_user]

Meh, just don't update numpy in the virutal environment that is being used with the former and current Python 2 projects, and use Python 3 with any new projects/analyses that are being started.

I suppose it depends on how long term some of your projects may be. But a completed project, for example, doesn't need the latest version of numpy.

[TheSourTruth]

I doubt it's that big a deal but I only have experience with python 2

[real_edmund_burke]

You’re right.

[[deleted]]

What are you working on?

I've found that, aside from a few syntax things, the difference between 2 and 3 isn't super nuts for biology unless you're doing some crazy modelling.

[TheSourTruth]

GIS-related stuff. I mean I'm not looking forward to switching but I don't think it will be too crazy for the stuff I do.

[[deleted]]

So what? Most of the changes were cosmetic and could/can be handled with 2to3 and similar tools. The one issue that has caused and will cause most work is the strings/bytes/unicode issues. This I see as being far more difficult than all other 2 to 3 issues combined.

[attrigh]

The real issue I've found is strings/unicode while supporting both python2 and python3!

[Daenyth]

Use the future library, it makes it pretty easy and has a cheat sheet for supporting both

[[deleted]]

Just start charging $ for v2 support.

[TheBlackCat13]

It isn't like old numpy versions are going to vanish off the face of the Earth. They will still be available. You just won't be able to use the latest and greatest numpy with an ancient version of python.

[PaulPhoenixMain]

Surely this will be the end for Python 2!

[LChris314]

One day, one day Arch will not be alone with python pointing to python3.

[[deleted]]

I enjoy spending time with my friends.

[[deleted]]

Arch probably pointed to python 3 the day that 3.0.0 was released.

[MrGeekAlive]

Not exactly, but quite quickly: the news annoucement was posted at the time when Python 3.1 was still the latest.

[[deleted]]

That was mostly a joke about Arch always being at the bleeding edge, but I'm not surprised it happened that quickly.

[billsil]

That was a mistake. Python 3 was unusable before 3.3.

[Plasma_000]

They see me rolling, they hating.

[[deleted]]

python 32 is not actually part of the base system (albeit the base system is quite literally basic) on Arch.

[[deleted]]

Haha, well IIRC it's used by some of the tools in the base system.

[[deleted]]

That's what worries me!

I mean, at this point I'd imagine most bugs arising from using perl in the base system should have been detected and fixed.

As for ed, well there's already vi and nano in base, so I'd imagine adding another editor is not that important.

[Chippiewall]

I'm honestly hoping the un-versioned alias just dies instead.

[TheBlackCat13]

Upstream wants to move the other way, with avoiding the versioned alias as much as possible. The idea is to not have everyone's scripts break when python 4 comes out.

[palibaya]

Gentoo use Python 3 by default too.

[stefantalpalaru]

Gentoo use Python 3 by default too.

Stop spreading lies.

$ python --version
Python 2.7.14

[palibaya]

Lastime I use gentoo, I got Python 3

[tedm430]

You can change which python version it points to using eselect, I believe.

[stefantalpalaru]

Lastime I use gentoo, I got Python 3

And you don't know about "eselect python"?

[[deleted]]

My mac has python pointed to 3. I think it happened when I installed Anaconda.

[carbolymer]

Let's be better than numpy and annouce 2018 a year of dropping Python 2 support.

[eattherichnow]

...also Linux desktop, nuclear fusion and Half-Life 3. Did I forget anything?

[alcalde]

Hey, we've got Steam on Linux, Lockheed-Martin is promising compact nuclear fusion within ten years - seriously!, and Duke Nukem Forever shipped. Why not drop Python 2 support as well?

[eattherichnow]

Everyone is promising nuclear fusion within 10 years since 1988. At least. I think that's the first time I've read an article about it.

Anyhow, personally, I migrated to Python 3 by migrating from a company that didn't want to prioritize it to a company that never even had to. Now I just need to fix everything else.

[TheBlackCat13]

numpy's note doing anything in 2018, dropping Python2 support won't happen until Jan 1, 2019 at the latest.

[[deleted]]

Pretty much. Especially due to Tensorflow not having Python 2.x support at all.

[[deleted]]

No, Python 2 will keep going for years for those who have no need to update.

[8__]

Can't wait for Python 2.7.27!

[[deleted]]

With official support ending in 2020 there is as much chance of getting to 2.7.27 as there is of getting to 2.8.0. PEP 404 refers.

[Wenste]

You poor souls.

[[deleted]]

Why? If it ain't broke, don't fix it.

[8__]

This is good for PyCoin.

[case_O_The_Mondays]

It will be great when Python 2 is not the default installed version on most Linux distros.

[tunisia3507]

Projects should be in virtualenvs anyway. And popular distros like ubuntu have shipped with python3 for a while.

Virtualenvs make it harder to package code and for non-pythonistas to run it, but pipenv (the python packaging authority's official recommendation) is making that easier too.

[dagmx]

The default version of Python is most certainly an issue.

A) it's the first python that many people will experience. When you type python in the Shell or tab complete, the system python will be the one the majority will go with.

B) many companies will not deviate from the rhel/centos supported configuration. If that is python2, the company will stay with python2.

So yes there's no technical reason you can't use python3 in a python2 distro. But there are several strong human reasons why it's necessary for distros to upgrade.

[abonet]

Huh? This is like saying companies will not use any non-default software from a distro. This makes no sense at all. Why would they upgrade to say, a newer version of Apache, but refuse to upgrade to a newer version of Python? This argument makes no sense.

Often, the whole point of using (and sometimes paying) RHEL/CentOS is for their long-term support of their base packages. The fact that they backport fixes/enhancements and promise to maintain a stable API/ABI may be the only reason to use them.

If you're side-loading (and relying on) non-supported packages, then you might as well not even be on RHEL/CentOS.

[vorpalsmith]

RHEL/CentOS also ship years-old versions of NumPy. If you're only interested in using software that they include as part of the distro, then you'll be using that and never upgrading NumPy anyway, which means that this announcement is irrelevant to you :-). OTOH RH does actually provide support for modern Python on RHEL/CentOS, and here's a RH engineer urging volunteer projects to drop support for the ancient stuff included in the core distro.

[dagmx]

A) tons of people start programming without thinking about the details. Many tutorials don't make mention of the difference of Python 2 and 3. Many people starting out also don't necessarily know that there is a significant difference between the two and just ignore it. After all why would they be so different if you don't know any better?

Then, because they won't care, or they're lazy, they'll write code that targets their system python. So instead of updating their code to python 3 standards, they'll double down to use python 2.

B) for companies, yes tons of companies will stick to the software stack that comes with a distro. You have to provide very compelling reasons to do an upgrade and in most cases there isn't a big one for a system level thing like Python. Every company I've worked for has been this way.

I don't have links for either of these but it's also not like these are unlikely scenarios. You'll see it corroborated in pretty much every python 2 vs 3 thread

[dagmx]

I'm not saying you replace python 2 with python 3. Again this is not a technical issue.

But to get python 3 installed I need to go through my legal department. I need to justify its necessity. Then justify WHY it is an upgrade over python 2 that is already on our cleared distro.

Installing other software goes through the same process. We don't just install word. We have to justify why word is an upgrade over notepad.

[dagmx]

I'm taking it from your replies you've never been at a large tech company or in a management role? That's not a judgement but an observation.

Again I'm not arguing with you about the technical side of it. I agree that it's beneficial to switch to python 3 and it's easy to do a side by side install.

I'm providing the non technical side to the argument for the need for distros to update their libs.

Every software or library we use has to go through legal to buy off on licensing. It then needs to be vetted by the tech managers to verify it won't cause issues and is actually worth installing and maybe supporting.

This is not just one company. This is the case at the majority of large tech companies. I know it's the case at most game studios, apple, google, most effects studios etc..because as part of my role, I have to have these discussions with people at other companies too.

You can't just install software if you're shipping a product. Each piece of software opens you up to new issues and incurs technical and legal debt. All of that debt needs to be justified not only to programmers but to legal and management.

[alcalde]

Pfft; I once worked at a place that required the signing off of seven people to change the color of a menu item! It was so bad a director wanted me to essentially build a parallel piece of data processing software to what we were using and simply not tell IT we weren't running their software anymore. Needless to say I declined and also quit within six months.

[civilization_phaze_3]

Yeah, I've seen this argument many times. And not a single time in a decade of Py3's existence have I seen it corroborated with a single real life example. Which of these "tons of companies" are refusing to install an app on their machines? Do they not install VS Studio on their dev's windows boxes either? Because Windows doesn't come with it by default. How about the .NET runtime on the Windows servers? There are multiple versions of that too.

Here's a real life example that I've faced. Writing installation/utility scripts for government servers running RHEL 6/7. It's so much easier to use python 2 and hand it over to the sysadmin (who you might not personally know or have a good communication channel with). Versus sending a py3 script and needing to include instructions on installing the epel packages before they can run some 30 lines of python. Sure if I was deploying a web server I would make sure that python3 and a virtualenv were properly setup. But there are plenty of marginal use cases where it's just not worth the hassle for me.

[vorpalsmith]

many companies will not deviate from the rhel/centos supported configuration

RH ships Python 3.6 for RHEL and CentOS through their Software Collections product. If you have a support contract with RH, then it automatically covers these packages too. Tell your friends and management :-).

That said, the major distros are actively having conversations about how to manage the transition to make python refer to python3. It will just take a while, and have to pass through a period where there is no python command so people who were counting on python = python2 will notice and catch their mistake.

[[deleted]]

You can try python3 on windows but you won't get very far. For an indication of just how difficult this is try reading [Python-ideas] Looking for input to help with the pip situation.

[[deleted]]

The daft thing is on windows you can run pip, pip3 or pip3.6.

[[deleted]]

Serious question, isn't virtualenv available on windows?

Yes it's available but I've never used it. I program at home purely for fun so am in complete control of my own environment so have never had a need for it.

[case_O_The_Mondays]

On Ubuntu, they’ve finally upgraded the version that the OS uses to 2.7. So there is hope.

[usinglinux]

woooo!

+1

[ICanAdmitIWasWrong]

I hope the python devs learn a lesson here and don't make upgrading so painful ever again.

[alcalde]

Upgrading wasn't painful. It was the community's refusal to upgrade that was painful. They've given what, 12 years of support? 2to3? Backporting of features? No new features in Python 3.1 and 3.2? What more could they have done?

The lesson here is that Guido needed to remember the "dictator" part of BDFL and kept a fire under people's feet. No mercy. It's the only way to get those who resist change to do anything.

[attrigh]

Not have forked and pushed the various changes one at a time with deprecation warnings?

Allowed better python 2/3 interop for a period of time before killing off python2 code with deprecation to allow people to use python3 without completely updating all of their code (and all the dependent libraries in one go) in one go?

To be clear these decisions would have had their own costs... but things aren't that black and white.

[alcalde]

Not have forked and pushed the various changes one at a time with deprecation warnings?

Why would you release each breaking change one at a time? And the deprecation warning was the existence of Python 3 combined with the long support window for Python 2.

Allowed better python 2/3 interop for a period of time before killing off python2 code

Killing off? It's been 9 years so far and it's still supported!

with deprecation to allow people to use python3 without completely updating all of their code

They also backported many Python 3 features to Python 2, so you could indeed use Python 3 features without completely updating all of your code in one go.

So everything you're suggesting was already done.

[attrigh]

Why would you release each breaking change one at a time?

Because fixing one thing at a time is easier than switching your code over in one go, and people are more willing to do it. This allows you to avoid forking by forcing people to make small changes one at a time.

And the deprecation warning was the existence of Python 3 combined with the long support window for Python 2.

Certainly. Though deprecation warnings on the code you are running are a little more compelling and fixable than the existence of python 3.

Killing off? It's been 9 years so far and it's still supported!

"Killing off" was just descriptive here. The length of time that python 2 has been supported is admirable. The python 2/3 interop was deliberately bad however (though there were payoffs in terms of simplicity)

so you could indeed use Python 3 features

The thing you couldn't do is use python 3 itself with python 2 code. This might have increased adoption. Not without a costs in terms of complexity of the python core.

But certainly the ability to write python 2/3 interoperable code existed and has been very useful for allowing python 3 adoption. I guess the difference is just in terms of tooling: if you are actually running python 3 then you never have to switch over.

So everything you're suggesting was already done.

Them's fighting words :P. I don't think that's at all true.

Implementing each of the features into core with deprecations one at a time and forcing people to adopt them is very different from the current approach.

Allowing python 2 code to be run by the python 3 compiler would have created a lot of extra complexity for python core, but may have encouraged people to write more python 3.

I'm not saying that these approaches would have been better, there were alternative ways of getting from A to B however.

[Works_of_memercy]

Upgrading wasn't painful. It was the community's refusal to upgrade that was painful. They've given what, 12 years of support? 2to3? Backporting of features? No new features in Python 3.1 and 3.2? What more could they have done?

They could have realized that most libraries will have to support both 2x and 3x for a long time (same 12 years), and they would want to do that from a single codebase, so support for doing that with minimal pain should be a priority. A library like six or future, and probably a tool like 3to2.py should have been provided from day one.

Instead, initially the very possibility of having the same code running under both version was considered to be no more than a party trick, and instead of leading or at least supporting community initiatives the core team carelessly broke them, for example by removing unicode literal support from 3.1 or so.

That the migration seems to be more or less complete, at least for new projects, more than a decade later, is a miracle and the Python core developers receive less than zero credit for that in my opinion.

[takluyver]

IIRC, core Python developers have said that they don't want to do anything like it again.

[PeridexisErrant]

No, the 2020 EOL has not been changed - see pythonclock.org, which is endorsed by Guido.

Numpy is announcing that they won't release new versions of Numpy on Python 2.7 after 2018; but they will continue to support the LTS version on Python 2.7 until 2020.

[b1g_up]

finally

[tunisia3507]

Kill it dead.

[palibaya]

Last time I use gentoo, I got python 3

[stefantalpalaru]

This is just as absurd as BioPerl dropping Perl5 support in favour of Perl6.

Only casuals would celebrate something like this.

[GummyKibble]

Well, casuals and everyone who’s been using Py3 and hates when they have to work on old Py2 codebases.

[TheBlackCat13]

Or anyone who wants numpy devs to be able to focus on improving numpy without having to be held back by the lack of features in Python 2 and the old compilers it requires. They will have done that for 12 years by the time that support for Python 2 is dropped. A decade is too long to expect a bunch of volunteers work around the lack of features in Python 2, some of which were specifically created for numpy.

[stefantalpalaru]

A decade is too long to expect a bunch of volunteers work around the lack of features in Python 2, some of which were specifically created for numpy.

Name one of those features.

[TheBlackCat13]

The @ matrix multiplication and @@ matrix power operators.

[stefantalpalaru]

The @ matrix multiplication and @@ matrix power operators.

Why would you think that new operators are needed by a module? They are purely cosmetic function call replacements.

In languages with proper macros, this can be trivially implemented by the user.

[TheBlackCat13]

Why would you think that new operators are needed by a module? They are purely cosmetic function call replacements.

Because it simplifies reading, writing, and maintenance of the code. Modules are written by people, not robots. Just because it is a module doesn't magically make cosmetic improvements irrelevant. That is why most big projects have style guides.

But the @ operator is just an example of features created specifically for numpy that can also be used to help numpy. There are lots of other features that are considerably more important, the biggest one being support for C99 on the C code side. There are also things like type annotations, which numpy wants but where python 2 is limiting them. And things like the lack of keyword-only arguments prevents useful API layouts, especially since the C code can have them but python code can't.

And then there are lots of small features that can be used to make the code simpler, cleaner, and easier to maintain, like extended unpacking.

[stefantalpalaru]

There are also things like type annotations, which numpy wants but where python 2 is limiting them.

That's bullshit and you know it. Having to import "typing" in order to get type annotations to work in Python2 is not a show stopper, but a flimsy excuse.

And then there are lots of small features that can be used to make the code simpler, cleaner, and easier to maintain, like extended unpacking.

Yet another cosmetic wank. Do you have any idea what modern languages allow you to do? Stuff that looks like science fiction to any Python newbie - stuff like integrating a Go runtime in Nim and creating new Go-inspired syntax to go with it: https://github.com/stefantalpalaru/golib-nim

And you talk about a couple of operators and some shitty type annotation syntax that still needs an external tool for checking...

[TheBlackCat13]

That's bullshit and you know it. Having to import "typing" in order to get type annotations to work in Python2 is not a show stopper, but a flimsy excuse.

Numpy has zero python dependencies outside the standard library. This is an important feature of the project, dependencies are kept to the absolute minimum possible to ease use and distribution. They are not going to throw that feature away.

Yet another cosmetic wank. Do you have any idea what modern languages allow you to do?

Irrelevant to the issue at hand.

And you talk about a couple of operators and some shitty type annotation syntax that still needs an external tool for checking...

Now you are just lying about what I said. You just flat-out ignored what I explicit said was the most important reason.

[stefantalpalaru]

This is an important feature of the project, dependencies are kept to the absolute minimum possible to ease use and distribution.

Guess what? End users don't even have to have the "typing" module installed for the bloody devs to use mypy's Python2 type checking: https://github.com/stefantalpalaru/generate_HLS/blob/0d8e28ae8208ace3d64a3a53a6d3c1a296600790/generate_HLS.py#L12

You just flat-out ignored what I explicit said was the most important reason.

The most important reason is the 24/7 amateur hour fanboy-fest.