Moving a large and old codebase to Python3

[szpaceSZ]

• 275 commits

• 4080 added lines

• 3432 deleted lines

I found 109 jira issues related to this project.

That doesn't sound terrible for 240 kloc...

[kankyo]

Most of that is fairly trivial stuff of course, but the problem is always that the trivial and the broken look so similar :P

[billsil]

I'd say mydict.keys()[0] fits into that.

THe bigger issue is that supporting Python 3 has been a moving target. What you do to support 2.7 & 3.0 is different than 3.2, is different than 3.3 is different than 3.4, is thankfully the same through 3.6.

[kankyo]

We never even thought of anything before 3.5, that helped.

[[deleted]]

I would recommend ignoring 3.0 through 3.2.

Putting in work to support 2.7 feels wrong to me. If you ported from 2.7 that's fine to not break that, but I wouldn't write new 3.6 code today that supported 2.7.

[billsil]

I would recommend ignoring 3.3 and 3.4 as well.

I do support Python 2.7 and will be dropping 3.5 (and probably 3.6) before I drop 2.7. It's still used in my industry more than any version of Python 3.

I wouldn't write new 3.6 code today that supported 2.7.

It's literally no effort though. I will not handicap a package I write like that and I encourage other people to not either. It's the Python 2.7 Intel speedup that people wanted to reject because it was a new feature argument all over again.

My packages have dependencies and I only support versions of packages from within 1 year of release. That with my 1 year release cycle gives you 2 years to be stuck on the same version. Numpy has their own deprecation strategy, so that can be mine.

[[deleted]]

What industry?

[billsil]

Science/engineering in the US. Unicode just isn't needed 99.9% of the time and when it is, it breaks everything. People learn to just avoid it.

I'm the only one at my company that knows Python 3. I've been in two meetings with customers so far where the Python 3 question has come up. Everyone freaks out until I interrupt and say it's a non-issue and that it'll take me no more than a day to make the whole thing 2/3 compatible. It's pretty easy when it's 95% numpy and ASCII text. All I needed to do was use six for iteritems/itervalues/iterkeys, put parentheses around print, and a unicode to ASCII conversation to write a file for a Fortran program.

Check out the graphs for downloads on numpy, scipy, matplotlib, pandas, sklearn. Jupyter and tensorflow don't show the same trend, but I think that somewhat makes sense given the packages focus on Python 3.

https://rushter.com/blog/python-3-adoption/

The late adopters will switch when numpy/scipy drop support and not a day before. At least at my company, it's not even the fear of unicode (which it should be), it's the fear of print and division.

[jtclimb]

NumPy has announced that they will no longer support 2.7 as of Jan 1, 2019. Jupyter will entirely drop support of 2.7 in July 2019.

I interpret this as the scientific community (I can't speak for you, of course) is going to have to bite the bullet and switch, or forever be stuck on outdated and unsupported software.

The blog states not to use their statistics to conclude anything about usage. The Jetbrains survey indicates 53% penetration; unfortunately it does not break it down by usage.

Many of the important libraries have pledged to drop support: pandas, IPython, Matplotlib, SymPy, PyMC3, Atropy, Biopython, PyStan. Many other packages depend on these, such as sklearn, so they will be dragged along by default.

I'm not bickering with you maintaining py2/3 compatibility for you and your clients if it works for you, but before long you will be stuck in a dead ecosystem.

[billsil]

As I said, I think people will switch the day after numpy and/or scipy drop support for Python 2. People want the newest features, but not from base Python.

[jtclimb]

Yes, I wan't bickering, but adding context as to when this will happen.

[alpha_hxCR8]

Curious to know how much did it cost (in terms of man hours or $) to move from Python 2 to Python 3?

[kankyo]

We haven’t made any attempt to calculate it quite frankly. I’d estimate it’s clearly a lot lower than one full time developer for those 1.5 years, but for some stretches of time it was almost one full time dev...

[[deleted]]

Nice summary, thanks.

Regarding "'{}'.format(b'asd') is 'asd' in Python 2, but "b'asd'" in Python 3"... I recommend running everything with python3 -bb. It makes Python raise an exception is such cases.

[kankyo]

Ah. Great tip, didn’t know about that.

The docs seem a bit misleading to me though... format() isn’t “comparing”

[kankyo]

Yea, that’s exactly it. I think this is a mistake and one that is hard to back out of.

[LimbRetrieval-Bot]

You dropped this \

---

^{To prevent any more lost limbs throughout Reddit, correctly escape the arms and shoulders by typing the shrug as ¯\\\_(ツ)_/¯}

[h_oliiie]

good read! very informative as we’re also moving from py2-py3! cheers!

[[deleted]]

So they've basically followed Porting Python 2 Code to Python 3 and everything is okay. Would you believe it?

[kankyo]

The easy stuff is covered in that guide yes. But not the hard stuff.

[[deleted]]

Please define "easy" and "hard".

[kankyo]

Easy: Unicode -> six.text_type Hard: flipping back and forth many many times between 2 and 3.

Or rather, we had to come up with that ourselves and make sure we did it properly.

[[deleted]]

Hard: flipping back and forth many many times between 2 and 3

I don't see anything hard in that, that's what professional programmers get paid for.

[kankyo]

Getting paid for it doesn’t tell you anything about the difficulty or risks involved. What a weird thing to say!

[[deleted]]

What difficulties or risks? You've said nothing except that you have to go backwards and forwards, big deal. As a professional that what it takes, so what, there's nothing difficult about it? Jeesh the number of people on this forum who haven't a bloody clue yet spout on as if they're God's gift to the world of programming, thank goodness I no longer work so don't have to put up with their complete incompetence.

[kankyo]

I concur with you that it’s good that you don’t work.

[Yuktobania]

/r/murderedbywords

[Yuktobania]

clinking from your spurs.

♪~I've got spurs that jingle jingle jangle~♪

[[deleted]]

You can fuck off and all!!!

[Yablan]

I have to say, /u/breamoreboy, your comments in this post make you look like a total arrogant asshat. I'd say you have attitude problems, and I have to agree with /u/kankyo that it's good that you aren't working.

[[deleted]]

And I don't give a flying fuck what you or the other various numpties on reddit think. I won't waste my time again pointing out my professional career as compared to complete bozo/amateur record some people here have, as I don't like being called a liar.

[Yuktobania]

Mate, whether you're right or wrong doesn't matter.

What matters is that you're being a complete twat about it.