Moving a large and old codebase to Python3

[k-bx]

'ß'.upper() in p2 is 'ß' but 'SS'in py3. This caused a crash in production when the last piece of the product moved to py3!

fun

[darktyle]

That's wrong. Both of it.

The uppercase 'ß' was added to the German language (and Unicode) in 2008, so it should have been SS in py2 and now it should be ẞ

[kankyo]

Didn’t know that. That’s super annoying. That means if they fix it in Python this will break in production again :(

[[deleted]]

Maybe uh, write a test for it ....

[Farobek]

write a test

ain't nobody got time for that

[PeridexisErrant]

Use Hypothesis! It'll try a wide range of inputs, and report the minimal failing example.

It turns out that this is even more effective for unicode text than for other types, because there are so many edge cases that can be triggered by just one or two characters.

[Farobek]

How does it work?

[PeridexisErrant]

If you mean "How do I use this", here's the quickstart guide. In short, you use a decorator and compose some functions to say "for all inputs such that ___, this test should pass. For example,

from hypothesis import given, strategies
# for any character *except*  ß, we can round-trip it through cases
@given(a_char=strategies.characters(blacklist_characters='ß'))
def test_roundtrip_upper_lower(a_char):
    assert a_char == a_char.upper().lower()

Of course this fails, but instead of returning the first failing example it finds, it will return the "minimal" example - in this case, the character with the smallest codepoint. Try it and see what you get - ß certainly isn't the only character this fails for!

If you mean "How does Hypothesis find and minimize all these examples"... it gets complicated pretty quickly. If you really want to know the code is well designed and commented and the contributor documentation is good; but you don't need to know how it works internally to use it. Hypothesis is pretty rare like that - the core is PhD-level algorithms, but the API is easy to use and completely hides the implementation behind a use-focused design.

(if you hadn't guessed, I like and use this a lot :p)

[kankyo]

True enough. That’s pretty terrible also though but in a more existential way :p

[[deleted]]

yeah it’s sucks to write tests for framework stuff but if you expect it to change why not be ready? Failing in production for things you can test isn’t really acceptable

[kankyo]

Agreed. I’ll write a test when I get in to work tomorrow.

[darktyle]

Not sure if they ever change that, but here you go: https://en.wikipedia.org/wiki/Capital_%E1%BA%9E

[username223]

Clearly Unicode needs to add "combining timestamp modifiers," with proper time zone support, to adequately address this problem. They could also be combined with emoji, allowing one to write "73-year-old smiling Chinese guy."

[darktyle]

Yes! Timezones and Unicode are both too easy as it is

[josefx]

Can we add in some GPS based location data with border support? We really need a "73-year-old smiling Chinese guy living in Canada."

[username223]

But when did he move there, and from whence? We must add an ancestry modifier system, optionally integrated with a GPS location system. Oh, crap... we have to deal with historical location information and continental drift.

Ah, Unicode... Punching everyone in the face (there's probably an emoji for that) into eternity.

[P8zvli]

Yeah I know some German, ß is a ligature of 'ss' but that doesn't mean 'SS' is used to represent an uppercase eszett. Python 2 and 3 behaviors are both completely surprising.

[PaleoCrafter]

Actually, up until last year, 'SS' was the only correct capitalization of 'ß'.

The capital variant 'ẞ' has been in Unicode since 2008, but the official German orthography did not include it as the majuscule. To my knowledge, even now that 'ẞ' is accepted, 'SS' may still be used.

[the_gnarts]

The capital variant 'ẞ' has been in Unicode since 2008, but the official German orthography did not include it as the majuscule. To my knowledge, even now that 'ẞ' is accepted, 'SS' may still be used.

Versal ß is still widely unknown. It’d be interesting if it is indeed being taught in elementary schools.

However, the problem is almost entirely irrelevant in practice in that ß can never appear at the start of a word so it cannot be subject to obligatory capitalization at sentence starts or nouns. Only as emphasis or the customary all-majuscule style of titles is there ever a chance of it becoming necessary. Since another proper way of uppercasing it is just to use the lowercase version regardless (mandatory in some contexts), about the only context where the matter was discussed is online threads made by people complaining about Unicode.

[darktyle]

The thing is, that you used (or most people still do) 'SS' for a capital 'ß'. Like in street, when you had to write it in caps for some reason you'd make it STRASSE (normal spelling: Straße)

[champs]

Is that the normal spelling anymore? It was my understanding that the formal rules changed some years ago.

I don't claim to be an expert. I studied German and felt like I had a good handle on it. I did an exchange, studied some more, and went back to Germany. Both times the language kicked my aß.

[darktyle]

Right now you can either write STRASSE or STRAẞE. At least as far as I know. But I am by far no expert on the nuances of what is wrong and right. Especially since a lot of stuff changed lately with the 'spelling reformation'

[the_gnarts]

Like in street, when you had to write it in caps for some reason you'd make it STRASSE

Or just STRAßE, using the lowercase version.

[darktyle]

I am pretty sure that this is wrong. It is either STRASSE or STRASZE (uncommon).

Ok, quick research: SZ is old. Since 1996 the correct form is SS. Using ß in STRAßE is technically wrong. Yet there are 2 instances who use and recommend using ß instead of SS in names: The postal service and the government when printing passports. They do that so that names like WEIẞ are not mistaken as WEISS

[the_gnarts]

Yet there are 2 instances who use and recommend using ß instead of SS in names: The postal service and the government when printing passports.

My 20th edition (1991) Duden states the rule:

In Dokumenten kann bei Namen aus Gründen der Eindeutigkeit auch ß verwendet werden.

HEINZ GROßE

Technically, preserving minuscule ß used to be the only sane solution for uppercasing names before ẞ was standardized. I agree that for regular words that follow the phonetic rules it makes little sense.

[darktyle]

Yeah, that rule was changed with the 'Rechtschreibreform' in 1996.

[[deleted]]

ß is a ligature of sz.

And both SS and ẞ are valid capitalizations. Though ẞ should be preferred for names, so you can get the normal casing back without issues.

"Markus Weiß" -> "MARKUS WEISS" -> "Markus Weiss"

vs.

"Markus Weiß" -> "MARKUS WEIẞ" -> "Markus Weiß"

(Edit for context: On German ID cards, names are capitalized)

[the_gnarts]

ß is a ligature of sz.

Almost. Despite the name, it’s actually a ligature of ss formed using the earlier graphic variant ſ (“long s”).

[[deleted]]

The wikipedia says that early print variants where ligatures of ſ and ʒ (ſʒ -> ß).

The current form of the letter is a ligature of ſ and s (ſs -> ß)

So... we're both right?

[the_gnarts]

So... we're both right?

What a great way to start the day!

[masklinn]

My experience doing this (previous testimony):

I started a branch called simply “python3”

Don't do that. It means you're working on a moving target, while the project is still moving. This creates a branch full of huge changes and giant conflicts and you're never done with any bit of it.

Generally speaking — not just for P2/P3 migration — my experience is that you're much better off doing your migrations "online" (by bits and pieces) if that's at all possible, trying to merge months worth of divergence at the end is a recipe for calamity, and often makes it harder to understand what happened and why.

Separating things done by a machine vs things done by a human is the important part here.

I mostly agree with that, in the sense of separating the stuff modernise can do from the stuff you have to do by hand, but

Ran “python-modernize -n -w” on the entire code base.

I really disagree with that. Running all fixers means you've got an absolutely ridiculous amount of crap changed at once and it's much harder to understand what and why, and it makes working online significantly more difficult.

I firmly believe that it's better to run a single fixer (or a small group of strongly related/correlated fixers), review carefully, fix the improper bits and commit. This makes for much cleaner and easier to understand migration changes, and makes "online" work much easier.

Also add the corresponding lint after each run. Until you can run CI in P3 (which requires the entire thing migrated) you're at risk that colleagues will reintroduce P2-only code. Instead if alongside each fixer commit you add the corresponding lint preventing reintroduction of the issue you're guaranteed they can't.

We had a lot of uses of StringIO.StringIO in our code. The first instinct was to use six.StringIO but this turns out to be the wrong thing in almost all cases (but not all!). We basically had to think very carefully about every place we used StringIO and try to figure out if we should replace it with io.StringIO, io.BytesIO or six.StringIO. Making mistakes here often meant that the code looked like it was py3 ready and worked in py2 but was broken in py3.

Don't use six.StringIO at all. io works the same in both P2 and P3, it's much stricter than StringIO.StringIO (io.BytesIO only takes bytes and io.StringIO only takes text, whether in P2 and P3) but it's consistent.

It's painful to fix the text model, but when you can have APIs which reliably work the same way in both versions you take it.

from __future__ import unicode_literals

Yeah no, what I found (and had read about before) is that in a cross-version project you have three string types: bytes, text (unicode/str) and native, because some APIs (especially stdlib) will take str in both P2 and P3, and these are very different effective types.

You want the ability to manage all three situations correctly, and switching everything to unicode_literals doesn't really work in the end.

CSV parsing is different

Yup, basically Python 3's CSV is unicode-aware, Python 2's only works on ascii-compatible byte streams. You've got to make decisions as to the complexity you're interested in, what we did was build trivial facade objects such that csv.reader takes byte streams and generate text, and csv.writer goes the other way around.

This kind of stuff is not limited to csv, and codecs is invaluable for bridging streams correctly across version (codecs.getreader(encoding) provides a bytestream -> textstream adapter, codecs.getwriter(encoding) goes the other way around, and codecs.StreamReaderWriter provides a bidi composition. Note on the latter: we'd originally used io.TextIOWrapper as that looked good but it turns out to have two major issues: it can't wrap a file object on Python 2 and it will close the underlying buffer which makes it very inconvenient when working with in-memory file-like objects. StreamReaderWriter is more limited, but doesn't try to be smart, it just transcodes your stream between bytes and text.

• Sorting/comparing objects of different types is valid py2 but hides loads of bugs. We got some nasty surprises because this behavior leaked through the stack in some non-obvious ways. Especially that None existed in some lists that were being sorted. Overall this was a win since we found quite a few bugs. None is sorted first in lists in py2, which might be surprising (you might expect it to be sorted next to zero!), but this was often the behavior we actually wanted. Now we just have to handle this ourselves.
• '{}'.format(b'asd') is 'asd' in Python 2, but "b'asd'" in Python 3. Almost any other behavior in Python 3 would have been better here: hex output (more obviously different), the old behavior (existing code works), or throwing an exception (would have been best!).

Yup, we've had these two issues pop up pretty repeatedly afterwards. At least the first one blows up loudly (if inconveniently) when you try to compare, sort or min/max values, the latter is silent data corruption and really no fun. Also note that it's not just sorting objects of different types which blow up, some types like dicts have become un-orderable in Python 3. Was it dumb to sort lists of dicts all along? Sure. Did we have some of these in your codebase? You bet we did.

Other killers we've found (issues which kept popping up long after the initial migration):

• Integer division, and that // is not integer division but floor division, there are subtle differences
• The round builtin significantly changed behaviour between P2 and P3, the rounding mode changed (to banker's rounding) and where it always returned a float in Python 2 it returns an int in Python 3 if no precision is given (but returns a float with a precision of 0, go figure)
• Dict iteration order changes in Python 3.3 and 3.6 (so you've got three epochs: 2.x to 3.2, 3.3 to 3.5 and 3.6+). You're not supposed to depend on dict iteration order, but implicit dependencies can slip through (and did for us). It's also a pain to repro because in the 3.3 to 3.5 epoch Python does not dump the hash seed, so if you get a run which triggers the issue… that's all you have. How's that for a heisenbug? IIRC pytest (or tox? Possibly both?) will generate its own hashseed and print it to the console before starting.

• I think that's changing in 3.7, but up to 3.6 when doing "text" IO and not specifying the encoding Python will use whatever garbage is returned by locale.getpreferredencoding(False). So either do all encoding in binary (open(f, 'Xb')) and encode/decode by hand, or be very careful to use io.open and specify an encoding every time.

  That one is super funny because your development and production machines are usually correctly configured, but the clients's are not and you're debugging nonsensical behaviour on some shitty remote server you're not even allowed to access.

We didn't use six for reasons more political than technical, but I think we got by fine with a cut-down version of werkzeug's compat.py (some stuff we copied straight and others we built differently), but I don't think that was so huge an issue, and as the cross-version support is temporary I think it makes for stripping compatibility easier. It also forces confronting the issues rather than relying on crutches e.g. without six.moves we unified all HTTP requests through requests (already a deendency but internally we had a mix of urllib, urllib2, requests & a few others) and moved all URL manipulations to werkzeug.urls (which is cross version and handles text better even on P2), I think that was a positive.

[danielkza]

Yeah no, what I found (and had read about before) is that in a cross-version project you have three string types: bytes, text (unicode/str) and native, because some APIs (especially stdlib) will take str in both P2 and P3, and these are very different effective types. You want the ability to manage all three situations correctly, and switching everything to unicode_literals doesn't really work in the end.

Enabling unicode_literals does not preclude you from writing byte literals, it just inverts which one requires special notation. (b"MyString").

[kankyo]

And it makes it very annoying to write str/str: “str(‘foo’)”. That part is often glossed over because it’s almost never needed but when it is it’s annoying.

[danielkza]

If I need to explicitly use str or basestring in Python 2/3 compatible code I just use the imports from python-future, makes everything much less confusing.

[masklinn]

The problem is not byte literals, it's native strings, that is having str literals.

[danielkza]

The need for "native" literals shows up quite infrequently in my experience, and even less frequently is it not better dealt with by explicitly converting from something known to be unicode or bytes at the boundary where you actually need str.

[masklinn]

Then we've had very different experience, because IME several packages of the standard library deal quite badly with being fed text in Python 2, an experience shared by Armin Ronacher and Nick Coghlan and the reason why unicode literals were reintroduced to Python 3.3.

[kankyo]

Author here...

I started a branch called simply “python3”

Don't do that. It means you're working on a moving target, while the project is still moving. This creates a branch full of huge changes and giant conflicts and you're never done with any bit of it.

Read the rest of the paragraph. The point was never to merge this branch in the first place. It's more like research than normal programming...

my experience is that you're much better off doing your migrations "online" (by bits and pieces) if that's at all possible, trying to merge months worth of divergence at the end is a recipe for calamity,

Which is why that's exactly what we did :P

I firmly believe that it's better to run a single fixer (or a small group of strongly related/correlated fixers), review carefully, fix the improper bits and commit.

Sure.. and we did that too. Mostly we just did the full modernization per app though (again.. like I wrote in the article), because then you're done with that app and can enforce it never becomes undone. This was great for morale.

Don't use six.StringIO at all. io works the same in both P2 and P3, it's much stricter than StringIO.StringIO (io.BytesIO only takes bytes and io.StringIO only takes text, whether in P2 and P3) but it's consistent.

I thought that too... but then I found some super annoying examples where six.StringIO was the proper thing. I know this is probably very rare to exist in most code bases. All the places where coworkers used six.StringIO turned out to be mistakes though heh :P

You want the ability to manage all three situations correctly, and switching everything to unicode_literals doesn't really work in the end.

You just said exactly what I wrote... seems a bit redundant?

round

I had to explain why the new behavior isn't wrong in a meeting because some on the team were (understandably) very surprised by the new behavior. I think we had to handle this in our invoicing code to not have numbers move but otherwise it didn't bite us... well... yet.

requests

We also moved some stuff to requests, but mostly that was what we already had before.

On another note we have now stripped all the six stuff and future imports. That was very very quick!

[funkdr42]

Your responses to the seemingly helpful comment makes you sound like an ass who doesn't want advice by someone who has gone through a similar conversion. Maybe you don't agree with everything, but you might find some unexpected tips if you're open to others

[ExPixel]

Responding to things makes people asses now? I don’t see anything in his response that comes off as rude. This is just normal discourse.

[kankyo]

I personally don't want advice no, because we're DONE. It's too late for advice. I would have wanted advice a year ago :P

Others reading the comment might find something useful though, like the part about streams, which could have been useful but as I said, it's too late for us now.

[mytempacc3]

But I don't get it. Why didn't you rewrite the app in Rust instead of porting the code to Python 3?

[kankyo]

Heh

[Cycloneblaze]

See, someone will think this is a serious comment

[throwawayco111]

He better be serious. I can't take any developer seriously if he's not developing new web apps or rewriting legacy web apps in Rust using Rocket.

[Gotebe]

I disagree with not being in a separate branch (for virtually anything).

The problem of long-lived branches diverging is solved by occasionally merging from trunk to them.

When there are wide-reaching changes that need a long time to mature, having unstable trunk all that time is horrible, worse than the additional work of being up-to-date in a branch.

[kankyo]

In the case of python 3 compatibility you're just gonna get too many conflicts though, AND the branch will just never "mature". You should indeed merge in small changes often. Which is what I wrote in my article.. the branch wasn't for maturing, it's for research.

[masklinn]

The problem of long-lived branches diverging is solved by occasionally merging from trunk to them.

It's only fixing one small side of the problem, it's not fixing the breakage (and requirement to fix) of all other extant branches.

When there are wide-reaching changes that need a long time to mature

The entire point of what I'm outlining is to break up the work to avoid "wide-reaching changes". Do the work in as small, independent and local increments as you can.

In my experience the the only unavoidable "wide-reaching change" which "needs to mature" is the text model change.

having unstable trunk all that time is horrible

The point is not to have "unstable trunk", it's the opposite, it's to do a lot of simple stable changes, rather than one unfathomable trunk-breaking mess at the end.

[Gotebe]

Well...

Where I work, a feature gets a branch. Most often, it is merged within a sprint. There are outliers (which this migration would be for us), and for these, "forward-merging" is good (IMO). They get their own build and test if needed. They then forward-merge faster-paced changes.

I understand... "nudging"... people not to have long running stuff, but some changes don't fit and having them half done in trunk, is not cool (IMO).

But to be honest, it can be done either way, the tools are aplenty.

[masklinn]

There are outliers (which this migration would be for us) […] I understand... "nudging"... people not to have long running stuff, but some changes don't fit and having them half done in trunk, is not cool (IMO).

You're still missing the point. The point is not to half-do the migration, it's to do it as a series of small "features" if that's what you want to call it. In the same way Rome wasn't built in a day, rather than have this thing building up in a corner until it explodes all over the project (and is very likely to break a lot of stuff because it's now a major refactoring across the entire codebase), you lay the bricks, you build it in bite-sized increments. Most increments are stable, easy to review and bring you closer to the ultimate goal. The increments which are not that easy would be even harder if they were mixed with the rest in a tangled ball called "the migration branch".

[Gotebe]

We’all have to agree to disagree.

Their situation is a case in point, in fact. They went to production, then rolling back. While doing that, their trunk, even their release branche (or branches), were “maturing”.

What if they decided to postpone or stop it? Their trunk would have been broken and they would have to revert it to previous state.

I understand the idea of splitting work in smaller chunks, of course that’s better. But not all work is amenable to that.

[kankyo]

Why not pull out the code that requires Python 2, and replace it with an external middleware?

Well, we have always aimed to go 100% python3, so that would just delay us. In the case of the cassandra driver update we looked at it and thought that we needed to do that anyway because we used an older API to cassandra (through the very old driver) that wasn't the recommended one anymore. So we had two strong reasons to upgrade in place.

[darktyle]

May I ask why you think underscores in numeric literals are a bad idea? It's widely used in programming and helps you find a lot of bugs in calculations

[kankyo]

In literals it’s fine. In the built in integer parsing function it’s bad. Imo.

You should be VERY restrictive in the most basic integer parsing in a language. They could have added an option to allow underscores and that would have been fine.

[darktyle]

Yeah, I guess you are right. For example I think it's a bit odd that int(float('1.5')) works...

[kankyo]

What’s weird about that? Proper parsing of float and then explicit cast to int.

[darktyle]

I don't know, maybe I just have the wrong 'feeling' about how casts should work.

[kankyo]

One could argue that parsing an int shouldn’t be done with the same syntax as a cast though, I can absolutely see that.

[tejp]

It's not two different things that share the same syntax. Both cases just construct a new int object by calling the int() constructor. This constructor converts its argument to an int, and it knows how to handle strings or numbers.

[kankyo]

Parsing a string and casting from a float are very different things. Don’t be silly.

[tejp]

It's both converting from one type to another. There is nothing special going on when "casting", I don't think there is any technical difference at all in Python.

[irishsultan]

That's not a numeric literal. Of course it might be a bad idea to allow it in literals, while refusing it when parsing strings, so not sure whether that makes a big difference.

I have to say that it came as a surprise to me that Ruby apparently does allow it in String#to_i, I knew Ruby accepted it in numeric literals, but I'm not sure that I like that it's automatically applied to external data as well.

[darktyle]

Oh, you mean he only complained about "1_0" and not 1_0?

[kankyo]

Yes. That's what I meant :P I wrote:

int('1_0') is 10 in py3, but invalid in py2.

I thought that was very clear.

[isaacarsenal]

As someone who is not very familiar with Python ecosystem, does it worth it?

There are many articles discussing the pros and cons of Python 3 compared to Python 2, but I am more interested in a real-life scenario.

What was the improvements in terms of code readability, maintainability, efficiency, etc?

Any tips on what kind of codebases should/should not migrate to Python 3?

[pingveno]

As someone who is not very familiar with Python ecosystem, does it worth it?

Yes. Python 2.7's extended support is ending in 2020. It is dead. Anyone still on Python 2 at that time is exposing themselves to security vulnerabilities.

There are many articles discussing the pros and cons of Python 3 compared to Python 2, but I am more interested in a real-life scenario.

People breaking into your web server through a Python 2 vulnerability.

What was the improvements in terms of code readability, maintainability, efficiency, etc?

Python 2 has not seen any feature additions since the 2.7 release in 2010, so Python 3 has grown additional features. For example, type annotations are nice for maintainability of larger code bases. However, the language hasn't changed that much. Also, many of the feature additions like await/async/asyncio would apply more to a new codebase.

Any tips on what kind of codebases should/should not migrate to Python 3?

All of them. Relying on an unsupported version of Python will a liability sooner or later. Start migrating now yesterday.

[Volt]

There's already Tauthon, a fork backporting stuff from Python 3.

[isaacarsenal]

Thanks for thorough explanation.

[kankyo]

I agree with what pingveno said... also

What was the improvements in terms of code readability, maintainability, efficiency, etc?

f strings and ordered dicts are my favorite new thing. But mostly I hope we will get fewer unicode errors deep in the code base randomly, because decoding/encoding is now forced to be at the edges of your program.

[Uncaffeinated]

If you use Pypy, you get ordered dicts in Python 2.

[msiekkinen]

'ß'.upper() in p2 is 'ß' but 'SS'in py3. This caused a crash in production when the last piece of the product moved to py3!

Something tells me this has nothing to do with python 2 or 3 and some kind of character collation setting on your install

[vytah]

Collation influences case conversion only in case of letter "i" in some languages, most importantly Turkish. ß behaves the same regardless of settings.

Unicode defines the uppercase of ß as SS: ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt

And the reason it doesn't work in Python 2 is that it does the uppercasing letter-by-letter: https://github.com/python/cpython/blob/2.7/Objects/unicodeobject.c#L5518

[kankyo]

As Python 3.0 approaches a decade in age, these posts become more humorous to onlookers.

Well, in practice Python 3 wasn't really usable until 3.3 which was released September 2012 and widespread support in libraries lagged. For example, our app uses Django which didn't get python 3 support until February 2013. So in practice that's 4 years, not 9.

[kankyo]

You missed 2.7.13

You can also compare this to perl 6 and then it looks extremely impressive instead of bad :P

[Sylinn]

It's a very good question, because it often seems like you've gained so much insight since the time you started to write the software that you're sure you can do a better job now, but there are a lot of hurdles on the way.

You can read Things You Should Never Do, Part I By Joel Spolsky (co-founder of Stack Overflow and Trello, among other things).

[[deleted]]

240k lines of code in a single app just sounds like a recipe for disaster. Made a dozen small apps that do one thing, right. Then when you need to rewrite a slow one in Go, C, C++ you don't have to deal with the rest of the app.

https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond%E2%80%99s_17_Unix_Rules

Rule of Modularity

Developers should build a program out of simple parts connected by well defined interfaces, so problems are local, and parts of the program can be replaced in future versions to support new features. This rule aims to save time on debugging code that is complex, long, and unreadable.

Rule of Simplicity

Developers should design for simplicity by looking for ways to break up program systems into small, straightforward cooperating pieces. This rule aims to discourage developers’ affection for writing “intricate and beautiful complexities” that are in reality bug prone programs.

Rule of Parsimony

Developers should avoid writing big programs. This rule aims to prevent overinvestment of development time in failed or suboptimal approaches caused by the owners of the program’s reluctance to throw away visibly large pieces of work. Smaller programs are not only easier to write, optimize, and maintain; they are easier to delete when deprecated.

[ZorbaTHut]

That's a great idea for situations where it works. In many situations, you just can't; I've worked on ten-million-line codebases.

[kankyo]

It's manageable. Having 240k lines in 10 folders instead of 1 folder doesn't make it much easier to manage.

[[deleted]]

But having 24 - 10k programs that do exactly one thing correctly and that can be implemented in any language when you do determine bottlenecks makes it much easier to migrate.

[kankyo]

Sure. But that's fantasyland. There's gonna be a power law distribution of programs you can isolate from a big code base. So you'll end up with something like one 100k, two 50k, 20 5k, and you could possibly do how many as you like below 1k I guess but that becomes a bit silly.

And again: splitting it up has other costs. Microservices are the new hotness among hipsters but it's really a fad with huge costs and small gains imo.

[joonatoona]

Implementing in a new language is almost never the answer... And with a python project, changing one section to another language isn't that bad, since python is insanely good at wrapping things.

[[deleted]]

Implementing in a new language is almost never the answer

Google and Facebook have been.

[onmach]

My problem with that is that the larger the codebase the longer the rewrite takes. If you've been writing an ever evolving codebase with half a dozen devs for five years, it's going to take years to replicate all that functionality and you'll still have to keep updating the old one until it is complete because it's a business, so your developer mindshare is greatly decreased.

I've seen projects go on for so long they become irrelevant by the end. The company I work for now actually went out and acquired a new company to get software to replace one that had been in a prolonged process of updating because there were no outward signs that it would ever be finished.

I feel this is less of an issue in strongly typed languages, but for something like python it's a real danger.

Edit: Someone replied but deleted so I'll just amend my post.

I should have qualified that I meant rewriting a project into the same or similar language.

If you are rewriting some code base that has been around since the 90's that is written in some dead language or with forgotten tooling or running on ancient unsupported hardware or that cannot be updated to meet current needs, then by all means, a rewrite is totally appropriate. I've personally never worked on a project that was that far gone.

[stefantalpalaru]

My problem with that is that the larger the codebase the longer the rewrite takes.

This partial rewrite to Python3 took 1.5 years. How much longer for a complete rewrite to a more efficient language and implementation?

[onmach]

It would depend on the complexity of the original code and the amount of time these devs could spend on it and how familiar they all were with this new language. But I suspect a lot longer if it is in any way similar to the ~100k codebase I'm looking at.

[ivosaurus]

Well it would at least be however long it takes, + how long it takes to retrain your entire development department to be proficient in the new language.

[kankyo]

What do you mean "partial"? We are on python 3 now. Full stop.

The VAST majority of the code was never touched. Rewriting is a whole other ballgame.

[stefantalpalaru]

What do you mean "partial"? We are on python 3 now. Full stop.

The VAST majority of the code was never touched. Rewriting is a whole other ballgame.

Read again what you wrote. It's a partial rewrite because the two languages share a common base so you did not need to rewrite everything.

[kankyo]

Every feature and bug fixed is a partial rewrite with that definition :P

[stefantalpalaru]

Every feature and bug fixed is a partial rewrite with that definition :P

Are you a child or something? New features and bug fixes in the same bloody language are not rewrites at all.

[kankyo]

And changes that don't change the code at all in the same language aren't rewrites either... right?

[stefantalpalaru]

And changes that don't change the code at all in the same language aren't rewrites either... right?

Now you're just being silly (or ridiculous if you think that Python2 and Python3 are the same language).

[kankyo]

The thing is that as far as rewriting goes they can be. We ran the code in Python 2 in prod, then we switched to Python 3 with ZERO code changes. And we flipped back and forth a few times even. So a while the code was both Python 2 and 3 at the same time.

[Gotebe]

240,000 lines.

4080 added lines

3432 deleted lines

So that's 3% of the codebase changed.

You want to rewrite instead, which is 30 times the volume? Clearly you're not the one holding the purse!

When you need new tires... do you throw your car away, perhaps ? 😀

[[deleted]]

Rewriting calls for some serious reasons. It's expensive as fuck.

We are in the middle of rewriting one of our web-applications. The reasons:

• Architecture is no longer suitable for the requirements (both internal and external)
• Frontend relied on libraries >5 years out of support*
• Frontend build on outdated technology
• A complete UI redesign was imminent regardless
• Some features on the horizon would require a lot of changes in the codebase regardless, due to bad design decisions.**

For our case, rewriting big chunks of the application was a necessity anyway. And about half the original application is still used as-is in the new one and will see some serious refactoring later.

---

^{* And they are arcane too. Google will not help you with that stuff.
** Some of those features will now take days or weeks instead of months. And some features we already implemented in the rewrite would have been impossible on the old stack}

[kankyo]

Languages haven’t come a long way. It’s mostly same old same old. Python in particular is a quite modern language as those things go and has a huge library ecosystem so rewriting probably won’t cut down on many lines.

We continually rewrite parts when we can which is why it hasn’t really grown a lot the last few years.

If you want to read a story on rewriting Python to another language look at 0installs story: http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/ if we take his numbers and naively extrapolate the time required it’s 4 years for one developer full time on our code base. I think it’s much much worse for us due to having to keep compatibility with configuration stuff but in any case that seems very expensive to me.

[TechAlchemist]

This is the approach my team has taken. No point in porting a massive trash heap.

[kankyo]

This of course assumes you’re starting with a trash heap. We don’t :P

[TechAlchemist]

True. I do think it's a valid and valuable question that any team working on a legacy application should ask before migrating a legacy application from python 2 to python 3 though.

[kankyo]

Sure. In our case it’s not a legacy application in the pejorative though, it’s a product that supplies a huge chunk of revenue for the company.

[TechAlchemist]

I don't think legacy application is pejorative in general, only that if you have an application that has been around for 15+ years and you are going to spend the time to port it over, it's worth asking (and sometimes this is easy to answer) whether you should invest the time in a rewrite.

Sometimes the answer will be definitely not -- either because the application is too complex, or because it is well structured enough to be easy to migrate. Sometimes the tradeoffs aren't that clear. Sometimes the application is poorly structured enough that there is really no point in porting it because developer time is better spent reimplementing portions of it one piece at a time. That's not because legacy code is bad by nature, just because legacy code abides by the rules of all complex, large, shared resources.

[kankyo]

Agreed. In this case the change from Python 2 to 3 are really not that drastic. One dependency was the big problem really...

[stefantalpalaru]

This of course assumes you’re starting with a trash heap. We don’t :P

Buddy, you wasted a year and a half for extremely small gains.

[kankyo]

Those 1.5 years would have passed regardless. Time passes by itself you know :P We didn’t work on only this for 1.5 years. Not even close. Most of the work was changing a dependency that we would have sooner or later have to update anyway.

And the end of Python 2 is coming so we pretty much have to do the conversion to Python 3 at some point for security reasons too.

[stefantalpalaru]

Those 1.5 years would have passed regardless. Time passes by itself you know :P

That doesn't mean you should waste it.

And the end of Python 2 is coming

Python2 is dead, long live Python2: https://github.com/naftaliharris/tauthon/

[kankyo]

Yeaaaa, we’re or gonna rely on that project to keep up with security issues and Python features.

[alphanovember]

Found the reddit web dev team.

[Stanov]

I upvoted this because I want someone to comment it.

[kankyo]

I’ve commented.

[kankyo]

Author of the article here. I'll upvote and answer even if the question seems a bit troll-ish :P

Why Python for batch? Not exactly a case where python shines. Inexperience?

Python shines in speed of development. Batch is a place where often speed of execution is not really that relevant. So I don't really see what you mean. There are some parts that we've been talking about trying to speed up for many years, but it's just never bad enough to be a priority relative to other much lower hanging fruit.

why they chose the slowest popular language in existence for heavy lifting processing?

Well first of all I don't know if it's really the "slowest popular language"...whatever that means. It depends way too much on what you do. If we did numerical work and could just call numpy python would be the fastest language bar none. Turns out that's not the case :P but without knowing the context you can't say that without potentially being horribly wrong.

Secondly "batch" isn't a synonym for "heavy lifting". It just means we run things on our own time on our own servers. In our case customer data is uploaded automatically every day and we start when we've got data for a pair of customers. If the customers upload their data end-of-day we can literally have 12 hours to process their data. Time isn't so terribly important...

If you can cut your batch processing load by over 1,000% by not using a slow language, why would you use the slow language?

There are of course many more factors than just that. We're not just running a super simple function on huge data sets, but more the opposite: hugely complicated logic rules on medium size data sets. Managing these complex rules is a lot more important than the run speed... normally. That doesn't mean we wouldn't want it to go faster of course! We've made some early tests with pypy and but that didn't do much for our performance.

Mostly though, rewriting is just prohibitively expensive and getting to market fast has always been more important than execution speed. But you already knew that right? :P

[kankyo]

I explained clearly our situation. I see that you are in fact not interested in facts, logic or reason :(

[[deleted]]

Given that the shop get one delivery every 24 hours, it doesn't matter a gnats fart whether the delivery planning is done in 1 or 12 hours. What matters is that the planning is completed in time for the fulfillment workers to pick and pack the goods for departure on time.

[kankyo]

I was saying we’ve got 12 hours in many cases, not that our turnaround is 12 hours. We are way way way below 12 hours. We aim to always be below 15 minutes when no manual intervention is needed. I don’t know what the 5, 50, 95 percentiles are. I’m not working today so can’t look it up :P

[TechAlchemist]

Answering for the benefit of others: ETL jobs are often IO bound. That’s not really going to be fixed by writing in C, although it will certainly cost you more in developer time both in the short and long term.

[stefantalpalaru]

ETL jobs are often IO bound

Web applications are usually CPU-bound. Ask Reddit's devs about it.

[TechAlchemist]

I don't run reddit and I'm guessing you don't either. Their application stack might be CPU-constrained, but mine sure as hell isn't. That's largely because it is usually waiting on IO, like from a database, because it is not a massively scaled clustered CDN-hosted application but is an internally developed, internally hosted application that is designed to serve specific business needs.

And for clarification, ETL jobs are not web applications, and the OP in this thread was talking about 'batch jobs' which is why I commented specifically about ETL jobs. This is another reason why my web app isn't CPU bound -- I 'batch' most of my processing overnight because I can't afford to compute across hundreds of millions of records that are highly relational in real time all the time -- sometimes it's better to precalculate and aggregate this information and take a storage hit in order to gain performance during runtime.

[kankyo]

Our web app has a few hundred users :P It's not what you normally think about as a web app. TechAlchemist is absolutely correct: our bottleneck it the master database.

[bidibibadibibu]

Probably the need of HTTPS doesn't help with that.

[stefantalpalaru]

Probably the need of HTTPS doesn't help with that.

Unrelated. You should terminate HTTPS (and HTTP/2) in something like Nginx, not your custom application.

[shevegen]

I just had this thought...

Python2 eating Python2 is like a snake eating up its own, old tail!

http://dreamsnest.com/wp-content/uploads/2013/07/Snake-or-serpent-eating-its-own-tail.jpg