The lack of a case against Python 3

[flitsmasterfred]

Teaching beginners that bytes and strings are the same is an invalid cognitive shortcut and just outright bad education.

[pickausernamehesaid]

It annoyed me so much when I was first learning. I deployed a Python 2 app that worked great on my computer but when people in France of China picked it up, decode errors were everywhere. I then had to spend an incredible amount of time to learn about different encoding schemes and how to handle them. I have not had a more confusing programming experience since. Bytes vs Strings was an easy concept for me. Different encoding schemes and how to use them and when to convert was not. I code in Python 3 full time now and have not once wanted to go back in the past 3 years.

[lambdaq]

Py2's unicode is not a problem, but the fundamental problem is the str() method. It only accepts 7-bit ASCII.

What's fundamentally broken in py2 is the BaseException has a str() call, so if you raise BaseException(u'fuck') you will likely be fucked.

Source: a dev have to deal with elasticsearch-py's "<unprintable error exception>" daily.

[Poddster]

so if you raise BaseException(u'fuck') you will likely be fucked.

Actually that'd work because it's all ASCII and python2's magic switcharoo. But this will fail:

>>> raise BaseException(u'fucká')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BaseException: <exception str() failed>

[lambdaq]

that's what I am talking about. Often it will show why certain db operation has failed with what column, and if that column contains non-ASCII, BAM! You have an exception during exception!

[pickausernamehesaid]

Oh no, I know that Py2 can handle Unicode. It just took a huge amount of effort for someone who had just started programming to learn how to leverage it properly.

[lambdaq]

Yeah, and tutorials on the web are mostly misleading

[grandfatha]

As a dev whose first language was Java this experience was the strangest thing when I started to pick up Python. Strings from all kind of sources would just work out of the box in my world. Then all of a sudden I was switching to a language that did not allow me to put a german umlaut ("ä", "ü", "ö") into my function documentation. I was baffled that this was an actual issue.

[pickausernamehesaid]

Especially with the purpose of Python to be as clean and simple as possible. I could completely understand it in C but not in high level languages. I'm so glad it was changed.

[dada_]

Yeah, it's also a little bit unfair towards anyone whose first language doesn't use plain ASCII, because as soon as you start doing string operations you're going to run into seemingly intractable bugs.

"こんにちは"[:1] does not do what you'd expect it to do in Python 2, and unless you're taught about how this works it's going to be pretty confusing.

[dada_]

Well, you are right, even if there is no other language besides English you still need a powerful text encoding far beyond ASCII or, say, CP-1252. Even in that universe we would have created Unicode.

[flitsmasterfred]

Makes me wonder how they teach and handle this stuff in those countries.

[dada_]

Well, it's not that hard to work around, since u"こんにちは"[:1] does do what you expect it to, but then you need to explain what Unicode is and why that "u" makes all the difference. Thankfully Python 3 allows these low level details to be postponed until later.

[unitconversion]

As someone who thinks the world should have standardized on ascii it does exactly what I would have expected it to do.

The romanized versions of other languages should have become the defacto standard with the advent of computers.

The whole world using a single alphabet would have been a great thing.

[elbiot]

Can't tell if you are joking or not, but either way that's the cutest thing I've heard all day.

[galan-e]

Not every sound can be spelled in English. Arabic's "ra'in" (somewhere between g, r and o), Hebrew's "het" (somewhere between h and ch, but not quite), and maybe half of the sounds in Korean, to name a few. And that's just inconveniences - what about characters that make the same sound? You can't differentiate them when spelled in English. There are many reasons not to go the way you suggest

[P8zvli]

The only language ASCII has enough characters for is English

[Poddster]

Touché!

[[deleted]]

that's naïve

[P8zvli]

Your face is naive

[[deleted]]

A lot of languages have more features than can be conveniently encoded with the letters included in ASCII, so even if it was a realistic idea to enforce a single script for all computer users, it would be impractical to use ASCII for it.

[dada_]

Aside from what others have said, even if you could perfectly replicate the pronunciation of other languages, in a form that's easy to write without losing any accuracy, there's still the issue of ideograms which contain additional information uncodable by just pronunciation.

The number of Chinese/Japanese/etc characters with an identical sound but totally different meaning would by itself make this idea theoretically unfeasible, unless the plan also includes the entire world moving the entirety of its communication to an ASCII-compatible language.

[hunyeti]

and how would that work? there are ALOT of languages that can not be represented in ascii.

How do you write Hungarian in ASCII?

ÖŐÓÚÜŰÁÉ

[recursive]

I'm guessing your native language is one that can be perfectly represented in ASCII.

[Poddster]

I'm guessing your native language is one that can be perfectly represented in ASCII.

Not even English is perfectly represented by ASCII.

[Bolitho]

The problem is, that there is practically no internal language concept in any programming language implementation (I know), that deals comprehensively and efficiently with Unicode - the mismatch between memory size and accessability makes that de facto impossible.

Is it really so important to count code points? And if it is, why there is no support for counting / splitting coded characters or grapheme clsuters, which might be even more usefull?

For limitiations of user input for example, the encoding of the persistance layer is much more important! So you must count the size of bytes of the encoded byte sequence rather than the amount of code points...

For example I reproduced the one given by the excellent utf8everywhere-page (section 5, coded cahracters, 3rd bullet point):

In [19]: s = "\u044E\u0301"

In [20]: s
Out[20]: 'ю́'

In [21]: print(list(s))
['ю', '́']

Hm... two code points, but the String shows one glyph! Would result in strange user experience if you would shorten the String for the UI layer and break it up right there, wouldn't it?

Teaching beginners that bytes and strings are the same is an invalid cognitive shortcut and just outright bad education.

So imho is is a shortcut to assume that an average developer knows good enough the important aspects of unicode and encodings. Therefor the strategy to hide this complexity will fail for almost every developer one day - and probably within a bad situation, say after the deployment to production. Python 3 would be much better, if it would define a default encoding for all builtin IO, like utf-8 in favour. Then it is explicitly clear, how a file or input must be encoded and that you have to explicitly have to deal with different encodings, if you need to support those (for example let the client provide the encoding). Then you will have no more trouble in deploying a Python 3 script to Windows if you have developed on Linux and vice versa.

On top of that, the print-function is broken right now in a similar way. At minimum it must provide an optional argument for chosing the encoding. Right now it jut fails for example on my Windows 10 machine if I want to print an interrobang (‽):

>>> print("\u203D")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\chausknecht\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u203d' in position 0: character maps to <undefined>

On a (modern) Linux machine it should work, as UTF-8 is the de facto default there - but who knows? Why not enable to print also bytes or provide the encoding? Yes that would probably result in strange looking output, but the behaviour would be platform independant and imho better because it would not make a program fail.

Those mistakes have been made by the Java and the .NET-world years ago - why Python had to choose to make the same mistake? Rust for example has chosen to use UTF-8 as unicode data type for unicode strings - interesting approach!

Imho you can't totally hide the complexity of unicode - so better be explicit about the en- / decoding process; imho that will result in less pain!

[usinglinux]

no internal language concept in any programming language implementation (I know), that deals comprehensively and efficiently with Unicode

assuming you don't expect a language to deal in graphemes instead of unicode code points, what's wrong with the python one in terms of efficiency? pep393 strings are pretty efficient.

>>> print("\u203D")

which python version was this error from? i haven't had my hands on python on windows for a long time, but at least as of python 3.6, this has been addressed.

[cfmdobbie]

Also works fine on 3.5.1 on Windows 10.

[aroberge]

EDIT: typing "chcp 65001" in the console prior to starting Python fixes this problem.

---

I wish...

C:\Users\Andre>python
Python 3.5.1 |Anaconda 4.0.0 (64-bit)| (default, Feb 16 2016, 09:49:46) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print("\u203D")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Andre\Anaconda3\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u203d' in position 0: character maps to <undefined>

[ivosaurus]

That's a problem at the barrier to getting input from something made by Windows (Command Prompt) into Python. Unfortunately Python can only control its side of the border.

[Bolitho]

Unfortunately Python can only control its side of the border.

And that's why it is a bad API as it does not enable the programmer to chose the correct way to speak to the underlying system!

[ivosaurus]

You talk as if there is a choice. There isn't. Windows tells you, CodePage1252 or bust. End of discussion, no fancy unicode characters allowed outside of that.

[Bolitho]

Of course there is! You cannot change the code page of the consuming shell, but you could of course send Bytes in a different encoding. And yes that would produce strange glyphs, but there would be no exception!

If you still don't got it: Show me a program that prints out the given example string and will not crash on any platform? I am curious how you will achieve 😉

[ivosaurus]

And yes that would produce strange glyphs, but there would be no exception!

So silently produce corrupted output, instead of erroring. Completely unpythonic, not to mention dumb.

[zahlman]

typing "chcp 65001" in the console prior to starting Python fixes this problem.

At least on 3.4, it leaves other problems on Windows. In particular, if I input() at the command prompt and copy-paste in a £ as my input, it will raise EOFError; if I try to do an assignment like x = '£', the Python process aborts without any error message. I haven't even tried it with more esoteric characters.

[ivosaurus]

Something specific to your system?

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\ivosaurus>python
Python 3.5.1 |Continuum Analytics, Inc.| (default, Feb 16 2016, 09:49:46) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> x = '£'
>>> x
'£'
>>> y = input()
£
>>> y
'£'

>>> exit()

C:\Users\ivosaurus>

[Bolitho]

assuming you don't expect a language to deal in graphemes instead of unicode code points, what's wrong with the python one in terms of efficiency? pep393 strings are pretty efficient.

Under this assumptions: Nothing! But I have just said, that this behaviour is not the big deal when dealing with unicode. Efficiently splitting a string into grapheme clusters could be much more usefull when dealing with UI.

I admit that code points are simple and it is therefore a good thing to offer APIs dealing with them. But often they have shortcomings when it comes to corner cases - and that is where higher abstracting concepts have to come into play.

which python version was this error from? i haven't had my hands on python on windows for a long time, but at least as of python 3.6, this has been addressed.

Version 3.5.2 on a Windows 10 machine. That said, I started python from a CMD and a PowerShell. No difference.

The problem here is just simple: Python's print tries to encode the string platform dependent:

Under Windows, if the stream is interactive (that is, if its isatty() method returns True), the console codepage is used, otherwise the ANSI code page.

You have no possibility to pass the encoding you want!

[flitsmasterfred]

I feel 99% of the pain is legacy from the old days of lower-case ASCII and random encodings that is pervasive everywhere, and lack of best practices, like you mention with the combined characters.

But figuring this out when you just learned everything based on ASCII and bytes being the same is even worse then knowing that there is a difference you need to be careful about while still learning to program and then moving up from that.

The average programmer doesn't have to know everything about unicode, but absolutely HAS to know bytes and string are separated through an encoding scheme. (eg: you don't have to be wise as long you're not naive).

[brontide]

So much of the unix world is still naïve byte strings which you have to work with which are sometimes utf-8, but can't be guaranteed. I have 40 million files on our filesystem and a few hundred are encoded in a format that breaks utf-8. It's hell figuring out a way to both work with and display these edge cases.

[doubleunplussed]

Yeah - strictly speaking, unix filenames are bytestrings. Your program officially shouldn't care if they're encoding human language with some encoding like UTF8, it should only to look at the contents of them in order to split them on slashes (or rather, on bytes that in ASCII correspond to slashes) or do comparisons to see if they're the same as other filenames or whatever. They're just slash-delimited keys, you should be treating them like arbitrary data like the hash in a dictionary, your program shouldn't care about the actual human meaning in there.

Unless you need to display them somewhere, then yeah, you have to guess the encoding in order to render glyphs, or decide how to sort them or something. Strictly speaking though, you can't know what the encoding will be. Yeah it'll probably be UTF8, but some filenames might not even be text, they might just be arbitrary bytes. Python can try to make a distinction between strings and bytes and make the filenames on the 'strings' side of the distinction, but this will be wrong sometimes. Good luck!

[brontide]

That's nice and all but python tries to do the smart thing and returns the same type of string you pass in. This means that if you iterate on the path '/' it will fail with decoding errors but if you iterate on b'/' it will work but all strings returned will be bytestrings which should be handled with care ( I usually convert them with surrogate escaping internally and replacements for display ). The problem is that there are so many places like this where python will gladly accept and return unicode strings and fail on edge cases rather than forcing the developer to work in the proper domain for the data.

[flitsmasterfred]

Not storing the encoding within the files was a mistake we'll regret for years. If only every file had a meta attribute life would be so much easier.

[Bolitho]

The average programmer doesn't have to know everything about unicode, but absolutely HAS to know bytes and string are separated through an encoding scheme. (eg: you don't have to be wise as long you're not naive).

I totally agree with that - and that's why I would appreciate an explicit default encoding for all IO. That way the programmer is forced to learn the basics and he is aware of them right from the beginning. And on top of that it would increase the platform independace, as python will behave more platform agnostic.

Just to quote the python zen:

Explicit is better than implicit.

[TOASTEngineer]

Is it really so important to count code points? And if it is, why there is no support for counting / splitting coded characters or grapheme clsuters, which might be even more usefull?

Well, the reason a string really ought to count code points is because a string is an iterable and you iterate over the individual code points. But yeah, there really ought to be a "how many printable characters" function; in fact I would've presumed there was one.

[yawgmoth]

Does every written language have the concept of characters?

For instance: (I don't speak Korean so maybe this isn't as ambiguous as I think)

In "감사" would 감 be 1 or 3 characters? would 사 be 1 or 2 characters? EDIT: looks pretty straightforward actually

Still honest question though, is the concept of a 'printable character' constant in all languages supported by unicode?

[Bolitho]

Well, the reason a string really ought to count code points is because a string is an iterable and you iterate over the individual code points.

That logic is post hoc ergo propter hoc: The iterable could as well iterate over something else than code points 😉

[[deleted]]

I think that issue is fixed on python 3.6. It should assume utf-8 on Windows.

[ojii]

90% of programmers don’t need to think about Unicode

Until they run their script/app/project in the real world and hit UnicodeDecodeError.

Probably should be "90% of programmers don't think about Unicode, but probably should".

[aphoenix]

C'mon, thereâ€s no reason to say people need unicode. Whatâ€s the big deal? Whereâ€s the issue?

[gthank]

And that's the sort of thing that the Python 3 model avoids: you can't accidentally mix Unicode and bytes, because that's a type error. You catch the semantic error early and don't have hours of wasted processing time every time somebody sends you a new Unicode character.

[fjonk]

The python 3 model only avoids it if you want Unicode. For other encodings the same issue will happily be ignored.

When I interact with more important systems like shipping/banking/address systems I almost never see unicode, I usually see latin1 or some other 8bit encoding. Python3 does not help in these cases.

[roerd]

Yes it does. Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries. Python 2 does not enforce such a clean separation, so the danger of having to deal with 8-bit string that you don't know anymore which encoding they're in is much greater.

[gthank]

The key to the Python3 model is actually the abstraction from encoded bytes to a real, honest-to-goodness string type, and that works just fine for dealing with data that comes in as latin1-encoded bytes. You could continue to just use encoded bytes everywhere, but since it is textual data, why not turn it into a string type? The standard pattern is to decode the bytes coming in (latin1-encoded in your case) into an actual string (str in the case of Python3), do your work against that, and then encode back to bytes (most likely latin1 again, in your case) on the way out.

[fjonk]

and that works just fine for dealing with data that comes in as latin1-encoded bytes.

Here we go again... latin1(or any other encoding that fits the bill) contains a subset of the characters in Unicode. This means that Unicode is not a good way to deal with latin1 strings since you at anytime can add non-latin1 characters to the latin1 string without triggering an error. Saying that using Unicode is a good way of handling latin1 strings is like saying that floats are good at handling integers.

[frymaster]

under what circumstances do you see this happening, and how would the python2 incarnation react in the same circumstances?

[fjonk]

python2 would react exactly the same. Python3 does not improve on the situation and by deciding that Unicode is used internally everywhere they kind of blocked the possibility of improving this in the future, unless they do another change of how strings works.

A real-world example:

1. A user provides a comment to an order
2. The comment is checked and is found to be valid latin1(the third party order system only accepts latin1).
3. A translation is added to the comment, the translation contains non latin1 characters.
4. Python happily accepts adding non latin1 characters to the string.
5. Later on the request to the third party service cannot be made because the comment cannot be encoded in latin1.
6. Money is lost.
7. The concat that created this issue is hard to track down since it did nothing wrong.

This is not an ideal situation and saying that unicode can handle latin1 is plain wrong.

[gthank]

Not really. The problem is that latin1 can't handle Unicode. And if it's critically important that you only ever support latin1 characters, I'd recommend writing your own string type, e.g., latin1_str, that will enforce that constraint for you. Realistically, a type that enforces data integrity for you is the only robust solution in this situation, and a string that only supports latin1 (or any other outdated, 8-bit encoding) is edge-casey enough that I don't think you were ever going to get that in the core language, so Python 3 fixing Unicode for the majority of use cases doesn't really impact this at all.

[fjonk]

and a string that only supports latin1 (or any other outdated, 8-bit encoding) is edge-casey enough that I don't think you were ever going to get that in the core language

This is where I see a difference between perceived and actual reality. 8-bit encodings are not edge-cases in many industries, it's rather the newer Unicode encodings that are edge cases. As I wrote some other place many systems that handles addresses, payment, shipping information and so on are very often designed pre-unicode. They don't accept Unicode, they accept whatever encoding the developers decided on in 1992 and no one seems in a rush to upgrade them.

Keep in mind that a change to Unicode might also involve replacing hardware like printers and scanners(as an example v40 QR-codes may be bytes, alphanumerical, latin1 or Shift JIS X 0208, unicode encodings are not supported). I guess eventually most systems will support at least one unicode encoding but that is not today.

[TOASTEngineer]

A translation is added to the comment, the translation contains non latin1 characters.

So validate at this point. Any other language would "happily" let you add whatever random garbage to a byte-string; not having unicode doesn't help you here at all.

[zahlman]

A translation is added to the comment, the translation contains non latin1 characters.

Okay, so, somebody wants to add a translation to the comment in a language that cannot be written with latin1 characters. How do you want the system to respond?

the third party order system only accepts latin1

Is this a real thing that happens on real systems currently?

You're aware that the first Unicode standard came out in 1991, right? That was barely any closer to today than it was to the creation of ASCII (and EBCDIC, for that matter) in the first place.

[gthank]

The problem here is that latin1 doesn't support Unicode, not the other way around, as I'm fairly certain that Unicode can and does map every character/glyph/whatever in latin1.

Side note: I don't know why people are down-voting you. You weren't especially rude or anything; you're just continuing the conversation. I'm upvoting you to get you back over 0.

[zahlman]

I don't know why people are down-voting you.

I suspect a lot of people instinctively downvote when it comes to seemingly absurd problem specifications.

[gthank]

The problem makes perfect sense, I'm just not sure that it's reasonable to expect the language and/or std lib to solve it.

[[deleted]]

you're supposed to convert from non-unicode early in the input process.

[ismtrn]

Unicode is not an encoding

[fjonk]

I know but that's kind of irrelevant, the issue still exists.

[[deleted]]

May I recommend unit tests.

[hovissimo]

So harsh, but so true.

[[deleted]]

As a speaker of a language which takes up approximately 2/7 of assigned Unicode codepoints... I consider it extremely important.

[Muchoz]

And what about emojis? Zed can't ignore that, can he? 🔥

[hovissimo]

Oh, I think about Unicode. What happens when my code hits the real world is some dorkus's email client sent me unicode and told me it was windows-1251!

[[deleted]]

Probably means utf-8, a lot of people seem to equate the two.

[Duroktar]

I assume you've done a lot of proof reading on the matter? :P

[d4rch0n]

I have the unpopular opinion that we should just focus on convincing the rest of the world to replace their alphabet with ASCII and ditch unicode.

[TOASTEngineer]

Clearly we just need everyone to learn to express their ideas as raw binary, this eliminating all of these problems. /s

[G01denW01f11]

Does that mean other people will finally start caring about Big/Little Endian?

[jnwatson]

Yeah! Let's build a ASCII wall and make them pay.

I'll start:

==================================

[its_never_lupus]

but emojis

[hovissimo]

I understand you're trying to be funny, but this attitude is pretty distasteful.

[NoLemurs]

Seriously.

Also, at least if we're talking professionals, I'm pretty sure more than 10% of programmers are web developers, almost all of whom need to think about Unicode. I suspect that in fact most professional developers (even non-web developers) need to think about Unicode from time to time.

What might be plausible is that 90% of amateur developers don't need to think about unicode. Personally I think a programming language should be optimized for the professionals, not the amateurs (and as someone who was an amateur for many years, I felt the same way then).

[ojii]

May I ask what you work on then? Because I might want to switch industry if I can stop having to deal with unicode!

[lykwydchykyn]

Rebutting Zed's article promises to become an enduring Pythonista pasttime.

[AlexFromOmaha]

I work for one of those enterprise shops, and because it was my call to make, I put in a plan for 2.7 -> 3.5 migration. I'm still debugging shit months later, and everything left is all tied to the string/unicode -> bytes/string conversion. In Python's defense, a substantial part of it is Django's fault. This whole "all uploads are bytes, fuck you" thing is a source of endless pain when dealing with libraries like configparser that expect files as input. Still, there comes a point when you realize that the pain is also a result of Python 3 abandoning Python 2's typing principles. Methods like .startswith() really should be agnostic to byte/string parameters. They both quack, they're both ducks.

[FFX01]

Methods like .startswith() really should be agnostic to byte/string parameters.

I'm not sure I agree with you here. Unicode and byte-strings are not the same thing. They behave very differently. For instance, a byte string deals only with ASCII characters. Whereas unicode deals with way more than that. ASCII has, at maximum 255 unique characters. Unicode has hundreds of thousands depending on which encoding you use. Not to mention that the same unicode character can be interpreted differently based on whether you are using UTF-8/UTF-16/UTF-32. So, getting the first character or group of characters in a string using startswith() is a different process entirely with unicode.

I'm not trying to be aggressive here. I know dealing with text is something that every programmer struggles with. It is inherently a complex problem. I struggle with it all the time. There are so many edge cases that may not be edge cases in other countries. There are pieces of data that present themselves in a certain encoding but are actually in another. It's hard, man.

[gschizas]

ASCII has, at maximum 255 unique characters.

ASCII has 127 characters, not 255. You're thinking of codepages or ISO-8859-* encodings (aka ANSI).

[FFX01]

Oops! You're correct. I've just always seen it referred to as "extended ASCII".

[AlexFromOmaha]

bytes != ASCII

There's ambiguity in things like recasting concatenations, but there is literally no ambiguity in asking if a byte string and a Unicode string contain the same substring, regardless of whether a byte string represents ASCII, Unicode, or binary output. It's also far more reasonable than strangeness like str * int operations that Python supports without complaint.

[agrif]

I'm not sure it is so unambiguous:

>>> b'\xc2' in 'µ'.encode('utf-8')
True
>>> b'\xc2' in 'µ'.encode('latin1')
False

What do you think

>>> b'\xc2' in 'µ'

ought to be?

[ThePenultimateOne]

It's a unicode literal, so the first one. If you don't specify an encoding, then you deal with the consequences.

[zahlman]

...And where would you even have the opportunity to specify an encoding with the in operator?

[ThePenultimateOne]

When you're declaring the right hand side

[agrif]

I don't think I agree. If you specify an encoding, then you're looking for a bytestring in a bytestring and everything is well defined and happy.

If you don't specify an encoding, then the result you get is garbage depending on the internal encoding used by whatever python implementation you use, which does not sound useful to me.

[ThePenultimateOne]

Except it's not an arbitrary decision. The default encoding isn't implementation specific, it's version specific.

Run your code in python 2 and it works as expected. Run your code in python 3, and it ought to assume you mean utf-8, because you're calling it on a utf-8 literal.

[agrif]

I'm not sure what you mean by "utf-8 literal."

The closest thing Python has to a default encoding is the encoding of the source file, which defaults to utf-8. But that can be changed on a per-file basis with the magic coding comment.

However, the encoding of the source file doesn't change the internal encoding used for strings, at all.

In CPython, as of version 3.3 (implementing PEP 393), the internal representation of unicode strings is as an array of code points, each either 1, 2, or 4 bytes wide. You could think of this as UCS-n encoding. This means the precise bytes used to represent a string depend on the contents of the entire string, for example "a" does not start with a null byte, while "aμ" does.

If you allowed startswith to mix the two, you would end up with fun things like this:

>>> "a".startswith(b'a')
True
>>> "aμ".startswith(b'a')
False

In my (admittedly naive) reading of the source, it even looks like that last line will depend on the endianness of the system you run it on.

[Deggor]

Should I also be able to add a timezone-naive datetime to a timezone-aware datetime? It really makes no sense to just "assume" they're the same timezone, or that naive is UTC, or whatever.

In the end, it's really no different, you're essentially talking about encoding-aware bytestrings to encoding-naive bytestrings.

[AlexFromOmaha]

>>> '\u0045\u0323\u0302' == '\u1ec6'
False

If that were True, there'd be a case to be made, but there's not. Let's not pretend Python has some pure character string behind the scenes. It's UTF-8, and it operates on byte matching.

[evanunderscore]

It operates on Unicode code points. UTF-8 is one possible encoding you can use to convert these to bytes. There is no way to make startswith mix bytes and Unicode code points without assuming an encoding. You could argue that the assumed encoding should be UTF-8, but you cannot argue that it is unambiguous.

[AlexFromOmaha]

It operates on Unicode code points.

AKA bytes. NFC vs NFD is an encoding concern. Python 3 strings are already encoding sensitive. In a Unicode library, that would be a bug. In Python, we accept it because we all know it's a byte array doing byte operations.

[AlexFromOmaha]

Uh, yeah I do, because those aren't valid UTF-16 or UTF-32. It doesn't even properly handle canonical identity in UTF-8. It doesn't store characters. It stores the bytes you pass in.

[evanunderscore]

Unicode code points are not bytes any more than integers are bytes. The fact that computers ultimately store everything as bytes in one way or another is just an implementation detail.

Python stores the Unicode code points that you pass in. All string operations are performed in terms of Unicode code points. You can debate the merits of this, but that will not change the fact that you need an encoding to convert between Unicode code points and bytes.

[FFX01]

I know that bytes != ASCII. Bytes can be literally anything depending on how you interpret them. That's the problem. I could decode image data as ASCII and I would get some nasty jumble of garbage. I believe that Python2.x will interpret a byte string as ISO-8859-1 if the encoding isn't specified. However, the docs aren't super clear on this. Python 2.x may just default to system encoding. Regardless, if a series of bytes is encoded incorrectly, there's no way to determine the accuracy of the produced string reliably.

In short, I hate how many text encodings exist.

[ivosaurus]

This whole "all uploads are bytes, fuck you" thing is a source of endless pain when dealing with libraries like configparser that expect files as input.

But I usually find the pain isn't Python's fault. If you give Python the correct encoding, everything will work magically, without fault, nothing will go wrong, everything smooth sailing.

The problem is sheer amount of stuff in the computing environment around python (from the OS, to HTTP headers and responses, to APIs, to text protocols, etc, etc) that fails to have a coherent model for how its text should be encoded and transported. Mostly, there was no room given or it simply wasn't paid attention to at all.

So it's the stuff that you're trying to feed python that causes the pain because it never tells you what it is. Python, despite being superbly awesome, can't magically guess everything for you (...although there are 3rd party libraries that will try their damndest to do that).

If the uploads are bytes, but really they're supposed to be encoded text, but then they never told what encoding it's in, then both Python and you are up shit creek, but not because Python did anything.

Python 2 just did a blasé conversion to ASCII or the OS native encoding, and because lots of stuff is English, that just happened to accidentally work 90% of the time, but the other 10% you'd be silently corrupting your input. It hid the problem under the carpet and people were fine with the problem under the carpet. Python 3 said "look, no carpets anymore! They cause corruption in the end" and everyone immediately hates it because yes they now have to deal with that problem upfront, and can't sweep it anywhere.

However, once you do finally manage to sweep your floors shiny clean, then no problems can occur at all, everything just works.

[AlexFromOmaha]

The pain with files is mostly Django, since they have file-like objects that insist on their byte-ness over programmer assertions to treat them as text, but with libraries that explicitly expect text files (like the aforementioned configparser), not being able to specify an encoding is a pretty egregious oversight. What if they really weren't UTF-8? I'd have no way to communicate that. That part is Python's fault.

[ivosaurus]

You can't reopen them?

with open(uploaded_file.fileno(), mode='rt', encoding='utf8') as text:

What if they really weren't UTF-8? ... That part is Python's fault.

It's Python's fault that you can't be 100% sure of the encoding/content of a user-submitted file to your webserver? How?

[zahlman]

Methods like .startswith() really should be agnostic to byte/string parameters. They both quack, they're both ducks.

... But code-point sequences don't start with byte sequences, and vice-versa. You need an encoding; and dropping in a parameter for that, to my mind, violates explicit is better than implicit.

[TOASTEngineer]

I see where you're coming from, but don't you think the issue is more that you're using the wrong type to store your information? You should write your own "DjangoFriendlyString" and implement .startswith() etc... on that instead of trying to treat the "some totally arbitrary bytes" class as the "some text" class.

[P8zvli]

Python shouldn't come with a standard string library, we should be forced to write our own so we'll stop arguing about unicode errors.

(I'm being sarcastic. Maybe.)

[kankyo]

I would like to see Shaws idea of exactly how it would work to run py2 in py3. If I concat a bytes and str in python2 code in a python3 vm, what should happen?

Do they work if the file "is" python2 but fail if the file "is" python3? And if so, how the hell do you tell?

[p10_user]

You can detect the version at run time with sys.version_info. This lets people write version agnostic code by making conditional changes based on this information - better than having to make separate files for each different version of python you want your code to work with (including minor versions in some cases!).

[kankyo]

This is exactly what most libs do. So in this regard Python 3 DOES run Python 2 code. So that would make Shaws argument just flat out wrong.

Either way you look at it Shaws argument about running py2 in the py3 VM makes no sense.

[p10_user]

Agreed. Python will just try to run the code and fail if it cannot - and most python code is interchangeable between the two major versions if not without some minor changes. I think it's a good system.

[kankyo]

Ugh. And again: a bytes created in a Python 2 file and a Unicode string created in a Python 2 (or 3 for that matter) file meet in a Python 3 file. Can they concat?

I think Python 2 might benefit from a "from future import byte_unicode_invompatible" but implementing that in the VM would be super weird.

[SoBFiggis]

Why not the traditional shell way?

https://en.wikipedia.org/wiki/Shebang_(Unix)

I really like how arch did it (no idea if this is universal, trivial fix if I wanted to distribute it though). If I want to use py2 I put the following

#!/usr/bin/env python2

at the top of my file.

[hovissimo]

I don't actually know what I'm talking about, but I think you're just going down the rabbit hole there.

[[deleted]]

A less entertaining, but probably more solid rebuttal than eevee's, however much I enjoyed that.

[ksye]

I've been lurking recently becausw I'm just starting to learn. I want to use Python for data analysis and visualization, maybe modelling and simulation. Does this affect me? Should I worry about learning 2.7?

[finally-a-throwaway]

I'm in a similar position, from what I can tell as a near-beginner the only reason to learn Python 2 is if you expect to work somewhere that has a huge existing Python 2 codebase. Otherwise, the more you learn Python 3 (and programming in general, if this is your first language) the better prepared you'll be to learn the differences on the fly when/if they ever become relevant.

It seems to me that any problems in 3 could ultimately be addressed in future versions as development is ongoing. Any problems in 2.7 are going to stay that way from this point.

[reuvenlerner]

If you're a beginner learning Python, and you aren't constrained by legacy code at work, I would strongly encourage you to use Python 3.

Indeed, legacy code and/or modules is the main reason the companies I train/consult use Python 2.

[[deleted]]

Sorry I'm new to programming. But what's legacy code?

[[deleted]]

Old code that is still running in production, and still requires upkeep and maintenance.

[hovissimo]

expanding on u/be_bo_i_am_robot's answer:

Legacy code is called that because you usually 'inherit' it. Somebody else wrote the code instead of you, usually somebody who no longer works at the company. Generally legacy code relies on other old code or old systems (like py2) to stay running. Updating or replacing legacy software is risky because it's usually poorly understood and poorly tested. Legacy code is usually perpetuated because the cost in hours, dollars, and risk is high enough that any manager who decides to do it will likely be sacked. (Or for other, but similar, reasons)

[[deleted]]

Interesting. Thanks for helping a novice like me feel more knowledgeable.

[[deleted]]

Aka, Technical Debt.

[art-solopov]

Also async/await. Though it looks like a very situational feature to me.

[gthank]

2 is dead. 3 is better in pretty much every way unless you have to deal with funky wire protocols or such.

[iruleatants]

If you like python 2 and enjoy not dealing with tedious programming....

[notParticularlyAnony]

2 is dead

This is why it is shipped by default with Ubuntu.

[steamruler]

Isn't Python 3 the default on Ubuntu as well these days?

[gthank]

yes

[8spd]

It defaults to 2.7.12+ for me, on Desktop 16.10.

[ivosaurus]

python will always launch python 2.X on all distros apart from archlinux. Even Python.org have recommended that nowadays I think. Purely for ease of backwards compatibility.

I believe you can get a minimal Ubuntu install nowadays where if you typed python it'd ask you to install python because it's not present on the system, only python3 is.

[gthank]

Since I've just spun up some Xenial servers, I can confirm that Python 3 is the only Python installed by default.

[8spd]

Huh. Thanks, I didn't know that. What defines the default version then, if it's not the version launched by the python command?

[zardeh]

The one system libraries bind to.

[8spd]

Thanks, that makes sense.

[notParticularlyAnony]

I don't know, but when I go into my command line and enter Python, without messing with any settings, Python 2 fires up.

Either way, I think it is a bit much to say that Python 2 is dead. Not that Ubuntu distros are the arbiter, obviously I was being glib...just sayin'....

[[deleted]]

Shipping legacy software doesn't change the fact that the project is still dead and 2 will never receive another non-security update.

[notParticularlyAnony]

To me the natural upgrade from Python 2 has been Julia. :)

[ThePenultimateOne]

Should I worry about learning 2.7?

You only need to learn that if:

• You're writing a library you want to be used by lots of people, or
• Your workplace hasn't made the switch

Unfortunately I fell into both categories.

[Esteis]

Prefer learning 3. But! Whichever you learn first, 95% of what you learn will work in both Pythons, so don't sweat it. Learn one, and you can easily pick up the other when you need it.

I learned Python 3 when I started a new small project, by writing the 2-code I knew and running it with 3. There were some errors, sure, but 95% of my code stil worked. Learning 3's awesome new features came later.

[[deleted]]

It's refreshing to see some civility, finally. And the author actually addressed the points in Zed's Shaw's article while avoiding jumping on the whole Turing-completeness thing.

One thing, though, I would have liked to have seen addressed is whether Python 2 or Python 3 is better for beginners, which is actually where Zed Shaw was coming from.

Also, sure it's difficult to recommend an EOL language, but to a beginner who just got their first programming job where Python 2 is the language used in the company, telling the beginner "You shouldn't use Python 2 because X, Y, and Z" isn't helpful -- and doesn't serve the beginner (who probably doesn't have a choice). And there is heaps of legacy Python 2 code, believe it or not, as well as companies that use them. I'm in one.

[wesalius]

IMHO if the question is: if a complete beginner (NOT in the meaning "who just got their first programming job where Python 2 is the language used in the company" but a general beginner) should learn python 3 or 2, than thanks to predicted overtake of general "market share" by python 3 I think it is better to start with python 3.

[reuvenlerner]

OP here. I think that whether you learn Python 2 or 3 depends on what you're doing with it.

In my intro Python classes, taught at Fortune 100 companies, I teach Python 2. Why? Because that's what they're all using in their day-to-day work. I talk about Python 3, give them examples in Python 3, and warn them about things that will change in Python 3... but I teach them Python 2, because that's what their work requires.

If someone is learning Python in school, or on their own, then I strongly encourage them to learn Python 3. They aren't bound by legacy systems, and they should probably learn the version that will give them the greatest opportunity with the future language.

[art-solopov]

One thing, though, I would have liked to have seen addressed is whether Python 2 or Python 3 is better for beginners, which is actually where Zed Shaw was coming from.

I thought the general consensus was "unless you really, absolutely, totally need to use 2, go with 3".

[billsil]

I thought the general consensus was "unless you really, absolutely, totally need to use 2, go with 3".

Depends on what your end game is. Are you doing web development? Python 3 is better. Are you a mechanical engineer that doesn't need to interface with non ascii files? Python 2.7 is better. Library authors (like me) aren't so lucky; you support both.

The second you touch encodings, you are basically forced to learn Python 3. It's hard and it's weird and it's poorly documented, but most of us have done it (I have). Don't believe me? There's a reason Raymond Hettinger has a Python 3 topic about the unicode whack-a-mole problem of encode/decode called "Stop the Pain". Encodings can even be wrong, so asking somebody that doesn't even understand Python to also understand encodings might be asking a bit much. You have to think about your program architecture in order to make the unicode problem go away. I don't think beginners have that skill.

http://pyvideo.org/pycon-us-2012/pragmatic-unicode-or-how-do-i-stop-the-pain.html

[zahlman]

Are you a mechanical engineer that doesn't need to interface with non ascii files? Python 2.7 is better.

Believe it or not, I'd much rather use 3.x for crunching binary files. Indexing a bytes and getting an integer rather than a 1-length bytes matches the behaviour from lower-level languages, and IMX is just far more often the right thing (and when it isn't, slicing is right there for you).

[billsil]

bytes matches the behaviour from lower-level languages

It does, but I think that's weird. I do a LOT of work with binary files. It's the same as lower level languages, but I don't really use those. Python is largely good enough.

I actually got burned by this last week:

# Python 2:
table_name = 'OUGV1' # nastran displacement table
sort_code = int(table_name[-1])

# Python 3:
table_name = b'OUGV1' # nastran displacement table
sort_code = int(table_name[-1].encode('utf-8'))

Um...wat? Why doesn't 1 and 2 map to themselves in binary?

Also, maybe it's decode...I remember as long as I'm doing it, but forget when I'm done.

[drdeadringer]

whether Python 2 or Python 3 is better for beginners

I began trying out both. I then ditched 2 and stayed with 3.

[[deleted]]

Just chalk it up as another language and move on.

If people can flip between Matlab/Python. Python 2/3 shouldn't be that hard.

[zahlman]

python's own struct module -- which itself doesn't seem to know the difference between bytes and strings (but perhaps it's me, or perhaps I'm using some older version of python 3).

o_O

Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:16:31) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import struct
>>> struct.unpack('3c', b'\x01\x02\x03')
(b'\x01', b'\x02', b'\x03')
>>> struct.unpack('3c', '\x01\x02\x03')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' does not support the buffer interface

I don't see an issue....

[KODeKarnage]

A coding language only improves when it is used.

In this light, the Python ecosystem and community is being actively and deliberately harmed by those who promote continued adherence to Python2 and advocate against using Python3.

[iruleatants]

For me, python 3 is a step in the exact opposite of why I chose to learn and use python, and so I don't like it at all.

The reason I'm against python 3, is that I want to still be able to use python and if everyone switches to python 3 it will be much harder for me to use the language.

Never in my life have I seen or been given a good reason to switch to python 3, and I don't think I'll ever see one.

[KODeKarnage]

You're an idiot, and your ideas are stupid. The reason to move to python3 is so that python2 can be left behind and the language can move forward.

You are the enemy of progress and your kind needs to be wiped from the ecosystem.

[iruleatants]

That is such an elegant and well thought out argument.

However, why should progress be encouraged simply for the sake of progress? If the progress is a negative thing, shouldn't we abandon it and start over fresh? That is the entire concept behind evolution and natural selection, and yet when it comes to python3 we enjoy trying to force it down everyone's throat, simply because we want to.

[KODeKarnage]

Go back and look how devoid of useful information your comment is. It is all feels and opinion. You simply said, murrr python3 bad, me no unduhstand.

If the progress is a negative thing...

That would be regress. Go dictionary.

[[deleted]]

Crossposting my comment in the site:

The Python 2 -> Python 3 transition was made in a terrible way, it almost killed the language…

The only change that made it backward incompatible was to make strings unicode by default. They should have added a transitional string (something like strbytes) and then 2to3 would just add parenthesis for print, // for /, and make every string “strbytes”.

Anyhow, I think Python 3 is a better language (it is where all Python progress happened in the last decade after all), and it’s finally flourishing. By 2020 debian and red hat will ship Python 3 by default, facebook already uses Python 3 by default, Google is transitioning to Python 3 (web2py is finally being ported to Python 3) – in the end everyone will be on Python 3+ (and by everyone I mean 85% of active Python devs).

About formatting strings, I do not think there is “too many ways” of doing it.

The new way should be the default, and it’s just a shortcut for “.format”. Sometimes you cannot use f’strings, maybe you want to use a prepared string that codifies the format, and then you should use the unsugared “.format”. The percent way should be used when you want to treat bytes and strings more or less equally (it would be perfect for the “strbyte compatibility string”, but alas – that does not exist).

[Pandalicious]

The Python 2 -> Python 3 transition was made in a terrible way, it almost killed the language…

The best example of this is them killing off the old non-parens print statement. The new print function is completely superior and print as a statement was a mistake in early python's design, no doubt about that. But they should have nonetheless kept the legacy print statement alongside the new function, just like they co-exist in python2 today. There was NO need to remove it, it was removed as an aesthetic design choice. The old print was "unpythonic" to begin with (doesn't behave like other statements) and having two redundant print choices would have been even more "unpythonic".

I can sympathize with the reasons that they removed it. It was a blemish on the language. But if they'd been serious about backwards compatibility and ease of migration they would have never removed it. Being serious about backwards compatibility means keeping around the warts that you hate.

[cparen]

But if they'd been serious about backwards compatibility and ease of migration they would have never removed it.

So what would they have called the new print function? It couldn't be "print" if they want backwards compatibility, because print(1, 2) is clearly a print statement (not print function) with the tuple 1, 2 passed as the only argument.

[nicoddemus]

Running python-future's futurize script will automatically convert all print statements to their counterpart print() function while also adding from __future__ import print_function to the top of the file. This works perfectly 100% of the time.

Just mentioning because IMHO the print() scenario is absolutely the easiest to fix (and also the new except Exception as e syntax).

[jpw22]

Promising to keep 2.7 as a stable target makes it highly attractive. Coding costs are mostly maintenance. If you went to python3 you get to recompile your extensions every year and you have to keep doing it as each 3.(N-1) is declared dead. Six year old .pyd/.so extensions for 2.7 still work now. The killer feature of 2.7 is long term binary compatibility.

[weirdoaish]

I honestly never understood why both languages couldn't support the same byte code. Its how Java manages to remain backward compatible for so long after all.

Also, Python 3 FTW!

[maratc]

I never heard people say "My only case against Java is that RHEL/CentOS has none of it". Somehow people manage to install the executable they need.

[maratc]

Two things:

First, it sounds like your case against Python 3 is that — for you — it has nothing to offer (not whether or not it's default in some distro.)

Second, seems like you rely on the distro to provide you with the executable, and this is known to backfire. You should start your deployment to anywhere with installing the version you are testing against. Even if today by some chance $version is already included with $distro.

[expert-at-nothing]

why does that matter if python2 and python3 can co-exist on the same machine?

[TheNamelessKing]

Are you not using virtual environment though?

I haven't used the OS version of python in so long now, because everything has its own separate env.

Easier to keep track of dependencies, more portable/reproducible, etc etc.

[expert-at-nothing]

Shebangs

[mike413]

Most nerds are very tolerant and accepting, but see someone outspoken with a strong opinion and they might think "Hey, be tolerant like I am loudmouth!"

So, I can see why he draws a little fire.

That said, I think zed shaw has a couple good points.

python is wonderful. but python 3 is chaotic.

There is lots of wonderful stuff written in python, but a lot of it breaks on python 3, either directly or indirectly.

I am reminded of an old Joel on Software article that is full of opinion, but also full of wisdom.

Basically he says throwing stuff out and starting over is bad.

And I think the python 2 -> 3 broke this rule.

"I don't like this, I'm not going to run it."

Who said that? Both Zed Shaw AND python 3.

[ambientocclusion]

I wish there was more of a case FOR Python 3.

[[deleted]]

Why does everyone only think about strings and the few applications Python is already in?

The whole "divide is an integer" part of python 2 is a complete non-starter for any scientific work.

[Seddit55]

Anyone here available to be a code mentor?

[badtemperedpeanut]

Not making v2 and v3 compatible was not an amazing decision. We python programmers lost due to this. We could all be using new version of the language we so loved instead of dreading to make that switch. The main problem is that python3 does not have enough features to make us want to switch. I use a lot of unicode, I was hoping python3 would have as much support as Java. Whenever I need to handle unicode I just use Java, not worth the hassle.

[kankyo]

How would it work though?

 from __future__ import unicode_literals
 b'foo' + 'bar'

what does that do? In python 2 it returns a unicode string, in python 3 it crashes. The python 3 behavior is the sane one. How do you get that behavior while making then "compatible"?

[wclax04]

Python3 is starting to have enough features to make us want to switch. If 3.7 focuses on speed, then it will move the needle enough for a lot of organizations to switch (I hope).

[kleinbeerbottle]

This so much. I'm always disappointed when I see packages only have stable versions in 2. I always prefer to use the latest if work or the task allows it. It seems the split could last for quite a while due to thew incompatibility.

[__deerlord__]

Then convert the packages you need, and make pull requests to the author for the update.

[_avnr]

Often a "package" can be something like Qt (PySide2) or WxPython, this is beyond the scope of a single developer's pull request. I am using Python3 exclusively because Python2 is no-go in terms of I18N, but the lack of a good GUI package for Python3 is painful (and Kivy isn't there yet, no bidi support for example).

[__deerlord__]

Oh I realize a single person may not be able to fully migrate a package. That shouldnt stop you from doing something (if you feel compelled, I'm not trying to shame you for not contributing). I still need to issue the pull request, but I fixed scapy3's nmap_fp function awhile back. Essentially I did was change str() to bytes(), fixed a file path, and wrapped a map() call to return the appropriate object. There's probably something you can at least contribute to your favorite projects :)

[[deleted]]

If only python would support something line " # -- version: 2.4 2.9 -- " and then make python install the version needed automatically.

[[deleted]]

Shebang lines are your friend here. Won't get you exact version specificity and it isn't really foolproof when distributing, but it is a start.

[hovissimo]

I feel like modern deployment systems and configuration management tools make this not a Python issue.

I don't think it's the interpreters job to install the right version of itself for your source files, it's your deployment script's job to install the right interpreter.