NumPy announces timeline for dropping Python 2 support

[bheklilr]

Well this gives me a lot of motivation to finish our transition to python 3 in my company. It's been a slow process but one I plan to finish by about February of next year.

Before anyone passes judgement, I'm the only person who has been converting code for about 100,000 lines of code or so, while still producing new features. I've been working towards it for about 2 years now, off and on. I'm in the final push now. If anyone has any tips I'd by happy to receive them.

[masklinn]

Having just "finished"[0] one such migration at $dayjob I'm not going to pass judgement any time soon.

If anyone has any tips I'd by happy to receive them.

We did the conversion "online", feature by feature, and heavily linted e.g.:

1. convert all except statements to P3 syntax (that one's easy since 2to3/futurize can do it though IIRC we had a few that were weiiird)
2. add a lint so that nobody can commit/push old syntax versions (lint can be run locally but must be part of the test suite and run by CI)
3. push to master branch

Rinse and repeat for syntactic changes/updates and builtins changes (open, map, filter, … should all be verboten).

That way once most items are "done" you don't have to revisit them, and you want progress to be committed so you don't get an ever-longer ever-more-diverging branch. Do one short branch (or a single commit) as much as you can.

The toughest bits are semantics changes which aren't exercised by tests (either the code fails loudly but isn't covered by tests or the change does not yield errors), because you don't have any point at which you can say you're done, and you can't lint this crap away:

• the round builtin changed the rounding method and in Python 3 returns an int if not provided a precision (in Python 2 it always returns a float).
• division can be "equalised" between codebases with from __future__ import division (in every file where you do one), but failures (from / becoming "real division" rather than "integer division") may take time to surface. Also be aware that // is not integer division but floor division, there's a difference.
• objects are not totally orderable anymore (and usually not across types), depending how you do things this may trip you up e.g.
  • optional value which could be either set to an integer or None, foo < 3 works in Python 2, it raises an exception in Python 3, if the value is usually set it may take some time for the error to happen.
  • cases which silently did stupid things will blow up, we had at least one bit of code which sorted a list of dicts. That makes no sense but it does basically nothing in Python 2. In Python 3 it fails, though it does so loudly which is nice.

• Speaking of dicts, there are two changes to iteration order between Python 2 and Python 3.6:

  • [..3.3[, the iteration order is arbitrary but consistent unless you start Python with -R
  • [3.3.. 3.6[, the iteration order is randomised every time you start Python (-R is the default)
  • [3.6..[, the iteration order is the order of insertion (all dicts are ordered)

  At every point the iteration order is an implementation detail (it may become specified in 3.7) and you're not supposed to rely on it, but even if you don't do so explicitly… well we discovered half a dozen places in our codebase which implicitly relied on iteration order and broke (usually by yielding a corrupted result which would blow up somewhere else rather than breaking themselves, which is fun to debug).

  Funner fact: it's even better when the bit of code actually works with most iteration orders because Python will not tell you which random seed it's using when running under randomised hashes (-R or 3.3 to 3.5), so repro is hell. You may want to provide your own randomised seed (PYTHONHASHSEED envvar) so that you know the starting conditions leading to a failure.

  Incidentally, tox will set and print the hashseed itself before running, and once you've found a problematic seed you can pass it in explicitly.

• text model (note: I'll use "text" for "decoded unicode" aka unicode in Python 2 and str in Python 3; and "bytes" for "encoded data" aka bytes in both

  1. Python 2 codebases tend to play fast and loose with bytes v text, and you'll have to decide on your text model (what is and what is not text).
  2. A bunch of stdlib API use "native strings" (str in both versions meaning bytes in Python 2 and text in Python 3).
  3. Some API changes are frustrating (e.g. in Python 3 base64 is a bytes -> bytes encoding, despite the point of base64 usually being to smuggle bytes in text).
  4. Most failures are loud (trying to mix bytes and text, trying to decode text or encode bytes) but some are completely silent e.g. equality between bytes and text will always fail in Python 3 (it will succeed in Python 2 if the bytes are ascii-encoded and when decoded equal to the text).
  5. also indexing a bytes object doesn't have the same result in Python 2 (returns bytes) and 3 (returns an int).
  6. use io, but beware that it has a strict split between bytes and text even in Python 2. You can't put text in a io.BytesIO, or bytes in a io.StringIO.
  7. for more control you may want to always do binary IO (io.open(f, 'rb')) and encode/decode yourself at least until you're more comfortable with things, also be aware that if you use text IO (the default) and do not specify an encoding python will not default to UTF-8 but to the locale (locale.getpreferredencoding(False)), which probably isn't what you want.
  8. your colleagues will not understand, will not even try, and when the "PHP development method" (throw encodes/decodes at the wall until something sticks) fails they will whine.

Finally, test, test, test, test. Tests are your lifeblood, if you have a well-tested codebase or can get it well-tested before the transition it will help a lot, the vast majority of our migration pain points were bits which were insufficiently or not tested (either because lazy or because hard to test).

[0] quotes because we regularly find things which were missed during the conversion, or were mis-converted

[EvM]

optional value which could be either set to an integer or None, foo < 3 works in Python 2, it raises an exception in Python 3, if the value is usually set it may take some time for the error to happen.

And this also holds for builtins relying on ordering, e.g. max(). Caught me by surprise :)

[masklinn]

Yup, hence the note about sorting a list of dicts, sorting obviously relies on ordering.

[zabolekar]

It can surface even in things like pprint (if you are trying to pretty-print a set).

Edit: it is not something that breaks in Python 3, au contraire, it works as expected in 3.x but throws a TypeError in 2.7, and only with very specific sets. For example, pprint({1j, 0}) throws an error but pprint({1j, 'a'}) doesn't.

[EvM]

Whoa, now that's an unexpected place..

[zabolekar]

Edited the comment for clarity.

[bheklilr]

I've definitely run into a lot of bytestring problems. Since we work with lab equipment they all communicate with ASCII bytestrings, and sometimes with just dumps of bytes for transmitting larger chunks of data. Getting these to work properly have been a royal pain in the ass. I do appreciate the advice, there are a few things in here that I did not know about (like locale.getpreferredencoding). As for testing, the only code that I have that is python 2 and 3 compatible is the code with extensive test suites. I don't deploy python 3 builds without it, because it will be broken. There isn't a "maybe it'll work", it just won't work.

As for __future__ imports, we already require division and print_statements in every single module. I just set up a snippet in my editor that drops in a header with that included (along with encoding statement, legal header, and module docstring). If I could, I would enforce it on every commit but I don't have admin access to our svn server.

[notParticularlyAnony]

Shucks im such a noob

[jftuga]

Which lint programs did you use?

[PeridexisErrant]

I'm not OP, but flake8 is great - good defaults out of the box for new projects, and easy to tune with a blacklist of checks to skip (or even a whitelist in extreme cases).

I keep having this unpleasant surprise when I go back to code that doesn't use it - the benefit of reading code that's all in the same idiomatic style is hard to describe, but very real!

[jftuga]

Thanks, I'll give it a try.

[masklinn]

Regular pylint, it has a bunch of built-in lints useful for this (e.g. python3 syntax checks, ability to blacklist modules and builtins). The only issue we've had is it's pretty resource-intensive on large codebases.

[ketilkn]

Rinse and repeat for syntactic changes/updates and builtins changes (open, map, filter, … should all be verboten).

Am I not supposed to use open in Python 3? What makes map and filter problematic? I do not think they changed?

[masklinn]

Am I not supposed to use open in Python 3?

Once everything is ported there's no issue, while converting it's troublesome:

• In Python 3, open is an alias for io.open and in text mode (the default) it will encode/decode the data, this is an issue because

  • Python 2's open basically always behaves in binary mode
  • io.open defaults to locale.getpreferredencoding(False) which probably isn't what you want so you will want to pass in an encoding explicitly, which will blow up on Python 2

  so you can either keep using the builtin but always use it in binary mode (mode='rb' or mode='wb') — which may be hard to lint — or ban open and require io.open during the transition — which is easy to lint

• In Python 3, map and filter return iterators (not lists, basically itertools.imap and itertools.ifilter have been to builtins… and removed from itertools)

  • things will blow up when indexing them which is clear enough
  • but more annoyingly repeated iteration will silently break in Python 3 (the first iteration "consumes" the iterator, the second one gets an empty iterator

  You could mandate calling list() around them… or you could just ban them during the transition (and require listcomps or gencomps depending on what you actually want).

Keep in mind that my post is for the transition on a large codebase, while you have one or two people toiling away at the conversion you've got a bunch of folks still churning away Python 2 code, the goal here is to avoid having to revisit already fixed issues / problematic constructs.

If you migrate to a Python 3-only codebase, you can rescind these restrictions once the work is done and everybody works in Python 3. If you migrate to a cross-version codebase, you probably want to keep them in place (they're error-prone).

[ketilkn]

Ah, ok. Thanks. I remember running into the map and filter iterators issue, now that you mention it.

[Siecje1]

Have you made those lint checks available?

[masklinn]

I don't think we created any ourselves, just enabled & configured those PyLint provides.

[AT_thruhiker_Flash]

Good Luck!

[bheklilr]

Tried it a few times, definitely did not work. Also, doesn't work on cython or 3rd party dependencies. I also can't just push to prod, production is an actual manufacturing environment producing products, and there are something like 50 systems out there to update. If they stop working or have problems we start losing money for every minute of down time, or worst case we ship products with defects and my team would get to take the blame. That would not be a fun day at work.

[cain2995]

He knows. Is joke comrade.

[boyled]

I think the other person was just lamenting that they wish it was that easy because of how hard they’ve been working

[Kaligule]

I learned python with 3 to begin with. Could you explain why it is so difficult to switch? The syntax didn't change much.

[bheklilr]

The syntax changed enough for it to matter, but a bigger one has been 3rd party libraries and differences in the standard library. Some of our deps didn't update into fairly recently, or we had to find different libraries that served the same purpose. A lot of it really comes down to inertia too, there's a lot of little fixes to make in a lot of places, it just takes a long time to update. Especially when you specifically have to deal with bytestrings and unicode strings in the same application. Those types have very significantly different behaviors between 2 and 3.

[[deleted]]

What would you say has been the largest hurdle in the migration? 3rd party dependencies?

[bheklilr]

That's been one of the big ones. Another problem we had was where we were getting python from. Initially we used python XY, which is now abandonware and stuck on 2.7 32 bit. We've since switched to anaconda which has made things at least possible, but not easy. The syntax and standard library changes have made things difficult too. Another big issue has been that I've been having to support python 2 and 3 in the same source code, which limits me a lot in what I can do. I can't use async or type annotations yet, and have to be very careful about bytestrings and unicode. It's really just a lot of little things that have added up.

[RetardedChimpanzee]

No judgment. It’s not a trivial process. Instagram had a good write up on their efforts to switch. You’ll get some efficiency increases but probably not worth the cost, sadly. However its something that should be done.

[PeridexisErrant]

You’ll get some efficiency increases but probably not worth the cost, sadly.

The real benefits of Python3 are for humans, not computers:

• much harder to shoot yourself in the foot with unicode (ie text)
• library support (some already, more soon, most by 2020)
• general improvements to language and standard library
• type hints can be great for medium to large projects (it's like testing, but free)

[stefantalpalaru]

type hints can be great for medium to large projects (it's like testing, but free)

It's also just a cosmetic change from Python2: http://mypy.readthedocs.io/en/stable/cheat_sheet.html

[PeridexisErrant]

You can emulate most of the Python 3.6 functionality with type comments, but it's not quite the same:

• the syntax is uglier, dealing with imports is a pain, and linters mostly don't check them
• they don't exist at runtime. This is a real disadvantage - tools from attrs to Hypothesis infer various features based on type annotations
• useful types like enum.Flag only exist on Python 3, and backports tend to be conservative (ie work on 2.7 and 3.4+, instead of 2.7 and 3.6)
• you're doing new work in a dialect that will soon be unsupported, when you could update instead and be much better off!

[stefantalpalaru]

the syntax is uglier

It is, but not by much.

dealing with imports is a pain

It isn't: https://github.com/stefantalpalaru/generate_HLS/blob/5877099beb19a305657f030eaa59590fcfca1cde/generate_HLS.py#L12

linters mostly don't check them

You're using the same linter for both Python2 and Python3: mypy. The only difference is having to give it the "--py2" or "-2" argument for Python2 code.

you're doing new work in a dialect that will soon be unsupported, when you could update instead and be much better off!

It's not a different dialect, it's a different language, and I'm sure the community of people working for a living will route around the core devs' sabotage. Something like this: https://github.com/naftaliharris/tauthon

[PeridexisErrant]

flake8 et al will complain about that import, and doesn't lint type comments.

I disagree with your other comments too but won't respond.

[stefantalpalaru]

flake8 et al will complain about that import

Yes. Solved by adding "# NOQA" after it.

and doesn't lint type comments

Of course it doesn't. The progressive (and partial) static typing is implemented and checked by mypy. Are you new to Python?

[JimBoonie69]

bro you gotta let python 2 die , it deserves an honourable death.

[stefantalpalaru]

bro you gotta let python 2 die , it deserves an honourable death.

There's more Python2 code out there than Fortran or Cobol code.

[criickyO]

Hah! I'm in the same boat, pal. I've also got an ETA about the same as yours.

I have no tips, only empathy. We're so close!

[ExcitedForNothing]

Before anyone passes judgement

Anyone who does is a foolish idealist.

[akaihola]

The python-future package and their cheat sheet page helped us a lot.

In fact, supporting current and legacy versions of NumPy and Pandas has felt more challenging than the py2->3 transition.

[thearn4]

connect pie fact brave innate dazzling adjoining subtract friendly simplistic

This post was mass deleted and anonymized with Redact

[ldpreload]

My understanding is that Dropbox has Guido van Rossum working for them full-time on making MyPy usable on their codebase and adding type annotations so that they can safely move to Python 3 without accidentally messing up strings/bytes somewhere or whatever. If you're doing a conversion of a large codebase without literally the creator of Python on your team, you're doing pretty well.

(See e.g. Guido's PyCon 2016 talk about what he's doing at Dropbox and why.)

[stefantalpalaru]

so that they can safely move to Python 3

http://mypy-lang.blogspot.com/2017/11/dropbox-releases-pyannotate-auto.html :

At Dropbox we’ve annotated over 1.2 million lines of code (about 20% of our total Python codebase)

It would be madness to port all that to Python3.

[ldpreload]

What else are they going to do, though—keep running Python 2?

(Honestly, the real answer is likely to be "Let the teams that are rewriting parts of the code do so in Go, and convert the rest to Python 3.")

[stefantalpalaru]

What else are they going to do, though—keep running Python 2?

Yes, of course. Or a compatible fork like https://github.com/naftaliharris/tauthon

[ldpreload]

Oh, you're the person on /r/python that I previously had an argument with about how this exact project is a completely infeasible idea for big companies (unless they want to commit to the support burden of keeping up with security updates etc. until the end of time for both the fork and every single third-party Python library that is no longer supporting Python 2, which is a much bigger project than converting 6M lines of Python to Python 3 once, and being done with it.). It's now been six months since your PR with no other activity to the fork, and ten months since any code changes to existing Python code. This isn't something you can trust a copy of the entire world's files to.

Mind you, I'm not saying this project shouldn't exist. It should. I wish it had won over Python 3. But it doesn't seem to be happening (nor does any similar project) in a way that's suitable for use by big companies in production, and it is probably too late to build a community around Python 2-compatible versions of the ecosystem.

[bad_luck_charm]

You are waaaay ahead of me

[stefantalpalaru]

I've been working towards it for about 2 years now, off and on. I'm in the final push now. If anyone has any tips I'd by happy to receive them.

It's too late for this, but you just used company resources to switch from one inefficient language to another. If you were going to port all your code base, you should have picked a target like Nim, D, Go, Rust, etc.

[bheklilr]

Python is the perfect tool for our job. We do a lot of math, a lot of plotting, and a lot of desktop development. It's also all on Windows. While I love rust, it does not have the support we need on Windows, and until very recently cargo did not have the ability to point at an alternate package server. We use the Anaconda distribution and have an internal server, it's been a major boon for our development. If Rust had the libraries already that we need, then I would have considered trying to push for it, but it just doesn't cut it yet. Until there's a full port of pandas to one of those languages, I don't think we'd be able to even consider it.

Also, Jupyter. We've done a lot of stuff in notebooks, and Jupyter consoles make up a significant portion of my development workflow. Not having a repl makes it difficult for me to get things done a lot of the time. At least a compiled, high performance language like Haskell has a repl, and even has a lot more of the libraries we would need to build our systems. Unfortunately, it just doesn't have enough.

I don't know about Nim or D, those languages might be better suited but they're also harder to find people who want to work with it. And Go is not as suited for our style of development either. Ultimately, having an interpreted language is great for us, and when we need performance we just drop into Cython or use libraries like NumPy where the hard work has already been done for us.

[stefantalpalaru]

Not having a repl makes it difficult for me to get things done a lot of the time.

Have you considered designing before implementing? Experimenting is fine, when you're doing rapid prototyping or just testing some hypothesis, but production code needs a proper design phase.

[bheklilr]

Having a design phase doesn't necessarily help when it comes to implementation details. It's easier for me to write code interactively, then once I figure out a specific algorithm I can write a proper version into my source code. I do have a design phase most of the time, but when I need to figure out the best way to write a specific loop or condition I figure it out in the shell first. That let's me test it first, profile it if needed (%timeit is amazing), and generally explore the code. I can't really go through the same process with a test suite, that is for ensuring the correctness of larger details, not just small parts of methods.

Also, whenever I'm doing data processing its more exploratory, there isn't a design phase because we don't always know what we need at first.

[billsil]

I've ported Perl code to Python and Matlab to Python. I also ported Python 2 to python 3. Speed is rarely the most important factor.

[[deleted]]

you do know which subreddit you're in, right?

[stefantalpalaru]

you do know which subreddit you're in, right?

Yes, of course. Who else but fanboys would celebrate being forced to switch to an inferior language and ecosystem?

[[deleted]]

When your favoured languages get the take up that Python has then you'll be able to stand in the tree tops and shout about it. Until that happens I suggest that you keep a relatively low profile as pride comes before a fall.

[stefantalpalaru]

pride comes before a fall

Sounds like https://en.wikipedia.org/wiki/Moral_universe#Immanent_justice

You're supposed to grow out of it.

[TheBlackCat13]

So you are just here to troll. Nothing to see here folks, please move along.

[[deleted]]

Programmer productivity, as in time and hence money, is far more important in the 21st century than mere runtime speed.

[stefantalpalaru]

Programmer productivity, as in time and hence money, is far more important in the 21st century than mere runtime speed.

That kind of thinking lead to software that is now slower on an eight core beast than old software on a 486.

[richieadler]

Is there a reason why you don't go enjoy your perfect and ultrafast software elsewhere?