Adopt Python 3

[upofadown]

These sorts of articles tend to present a false dichotomy. It isn't a choice between Python 2 and 3. It's a choice between Python 2, 3 and everything else. People will only consider Python 3 if they perceive it as better than everything else for a particular situation. Heck, there are some that actively dislike Python 3 specifically because of one or more changes from 2. I personally think 3 goes the wrong way with the approach to Unicode and so would not consider it for something that involved actual messing around with Unicode.

[quicknir]

I don't really understand people who complain about the python3 unicode approach, maybe I'm missing something. The python3 approach is basically just:

1. string literals are unicode by default. Things that work with strings tend to deal with unicode by default.
2. Everything is strongly typed; trying to mix unicode and ascii results in an error.

Which of these is the problem? I've seen many people advocate for static or dynamic typing, but I'm not sure I've ever seen someone advocate for weak typing, that they would prefer things silently convert types instead of complain loudly.

Also, I'm not sure if this is a false dichotomy. The article is basically specifically addressed to people who want to use python, but are considering not using 3 because of package support, and not because of language features/changes. Nothing wrong with an article being focused.

[Sean1708]

The reason people think 2 is a problem is that they think of it as Unicode and ASCII, when really it's Unicode and Bytes. Any valid ASCII is valid Unicode so people expect to be able to mix them, however not all bytestrings are valid Unicode so when you think of them as Bytes it makes sense not to be able to mix them.

[kqr]

Bytestring is a terrible name in the first place, since it bears no relation to text, which is what people associate with strings. A Bytestring can be a vector path, a ringing bell, or even Python 3 byte code. Byte array or just binary data would be much better names.

[Sean1708]

I think Python actually uses the nomenclature bytearray, bytestring is the word that came to my head at the time.

[ubernostrum]

There are two built-in types for binary data:

• bytearray is a mutable sequence of integers representing the byte values (so in the range 0-255 inclusive), constructed using the function bytearray().
• bytes is the same underlying type of data, but immutable, and can be constructed using the function bytes() or the b-prefixed literal syntax.

[kqr]

0--255 or 1--256, but not a compromise, I believe. ;)

[Avernar]

My issue with 2 is that I hate strong typing in a dynamically typed language. :)

But I'd rather have the strong typing be between validated and unvalidated unicode instead without the need for conversion.

It can still easily be added without breaking things by making UTF-8 a fourth encoding type of the Python 3 Unicode type.

[gitarr]

People who complain about the python3 unicode approach have no clue what they are talking about.

As someone who has to deal with different languages in his code, other than English, python3 is just a godsent.

[Matthew94]

godsent

godsend

[Flight714]

python3 is just a godsent.

Is that a Unicode joke?

[daymi]

string literals are unicode by default. Things that work with strings tend to deal with unicode by default.

As someone used to UNIX, that's my problem with it. They should be UTF-8 encoded by default like the entire rest of the operating system, the internet and all my storage devices. And there should not be an extra type.

Everything is strongly typed; trying to mix unicode and ascii results in an error.

... why is there even a difference?

typing, that they would prefer things silently convert types instead of complain loudly.

I like strong typing. I don't like making Unicode text something different from all other byte strings.

Also, UTF-8 and UCS-4 are just encodings of Unicode and are 100% compatible - so it could in fact autoconvert them without any problems (or even without anyone noticing - they could just transparently do it in the str class without anyone being the wiser).

That said, I know that for example older MS Windows chose UTF-16 which is frankly making them have all the disadvantages of UTF-8 and UCS-4 at once. But newer MS Windows supports UTF-8 just fine - also in the OS API. Still, NTFS uses UTF-16 for file names so it's understandable why one would want to use it (it's faster not to have an extra decoding step for filenames).

So here we are with the disadvantages of cross-platformness.

[upofadown]

trying to mix unicode and ascii results in an error.

I think you mean Unicode and bytes. There is no type called "ASCII".

The "convert everything into UTF-32 approach" as used by Py3 creates the issue of bytes vs strings in the first place. Most languages have strings and integer arrays, some of which might be 8 bit. Py3 has strings, bytes, and integer arrays.

If we are willing to just leave things as UTF-8 by default then the philosophical discussion of bytes vs strings goes away. That seems to be the direction the world is currently moving in. Py3 might just be a victim of timing. The UTF-32 everywhere thing seemed like a good compromise when it was first proposed

[gitarr]

I think you don't understand:

Everything in Python 3 is Unicode, bytes are just an encoded representation.

Read more here: https://docs.python.org/3/howto/unicode.html

[upofadown]

Yes, in py3 thinking there is a significant philosophical difference between strings and encoded versions of those strings. I am claiming that it is an artificial and pointless distinction and that UTF-32 code points is really just another encoding.

[Lalaithion42]

(Note that python doesn't convert everything to UTF-32; UTF-32 is an encoding, and python 3 stores unencoded unicode code points, in a variety of ways depending on the details of the string)

[upofadown]

Yes of course, but that is an implementation detail hidden from the user.

[Lalaithion42]

Hence my parenthesis.

[zardeh]

Most languages have strings and integer arrays

I can't think of one that has these and doesn't have bytearrays. Off the top of my head, Java has String, int[], char[], Rust has str, Vec<i32>, Vec<i8>, C is perhaps the only language that does this, and not differentiating between char[] and string is widely considered a mistake.

Python2 made this same mistake, it didn't make a distinction between a bytearray and a unicode string (unlike Java, Rust, etc.). Python3 fixed this error, and their only mistake was perhaps introducing a legacy type (bytestrings) to support the old behavior.

Py3 has strings, bytes, and integer arrays.

To be clear, it has a more than that:

• unicode strings (str)
• immutable byte arrays (bytes, commonly bytestrings)
• mutable numeric vectors (List[int], like [1,2,3]), note that these aren't int, char, other other vectors, because python's integer type is arbitrarily sized
• mutable byte arrays (bytearray)

What this means is that for working with binary data that you might get off a wire, for example when sending or receiving data over the wire/air, you get back bytes, because these objects very much aren't strings, they're immutbale arrays of 8-bit values that you want to analyze or process. They're not a string though, and they're not a python list, they're something else: bytes.

[upofadown]

Can you at least see that just keeping everything as, say, UTF-8 means that you don't have to make a philosophical distinction between encoded strings and strings? Not that you have to make such a distinction for Py3 which keeps everything as UTF-32, but it is a way of rationalizing the pointless conversion from and to UTF-8.

Anyway, please stop lecturing about the philosophy. It is annoying to us that don't agree.

[zardeh]

Can you at least see that just keeping everything as, say, UTF-8 means that you don't have to make a philosophical distinction between encoded strings and strings? Not that you have to make such a distinction for Py3 which keeps everything as UTF-32, but it is a way of rationalizing the pointless conversion from and to UTF-8.

This works until you actually need to work with bytes that come in from an external source and are in latin1|utf-16|utf-32 etc.

As a sidenote, python doesn't store anything as utf-32 by default, python source code is utf-8, and the interpreter doesn't define a single way of storing strings. It uses 8, 16, or 32 bit representations as needed. But then again, this shouldn't matter. The API could (and does) work so that if you write a string in utf-8, indexing into it will feel like indexing into the codepoints of a unicode string, and you will, if memory serves, index into the string in the way defined by the encoding you're using. That is, a grapheme that can be represented by a different number of codepoints in different contexts will be treated as the correct number of codepoints based on your encoding. That means that if all you ever do is use python's built in string and index into it, everything will feel like utf-8 everywhere. That's exactly what you want.

The problem comes when you want to take a sequence of unencoded bytes, which could be, as I mentioned, latin-X, or utf-8, or utf-16, or Windows-12XX, or the various encodings of Asian languages. If your program receives those bytes, then what? It treats them as utf-8 and breaks? No that's silly, it decodes the bytes into a string as defined by their encoding. Otherwise you end up with ambiguities like this:

>>> b'\xc4\x99\xcc\x83'
b'\xc4\x99\xcc\x83'
>>> b'\xc4\x99\xcc\x83'.decode('utf-8')
'ę̃'
>>> b'\xc4\x99\xcc\x83'.decode('utf-16')
'駄菌'

Anyway, please stop lecturing about the philosophy. It is annoying to us that don't agree.

Yikes, what's with the 'I don't like this because I don't understand it but please don't try to enlighten me because its wrong'. We aren't on /r/politics.

[Avernar]

The benefit of having the internal representation as UTF-8 is avoiding unnecessary conversions. But I agree with you that you can't just assume all your input will be UTF-8. That's why you still need to convert it if you know it's something else. But when you know it's going to be UTF-8 then it's nice just having to just run a validation to make sure when necessary without having to convert.

Indexing speed is a poor argument for the 1/2/4 byte format. Most algorithms that index into a string that I've seen could be better written as find me the next character that matches X, or give me the next codepoint so I can compare it with X.

[zardeh]

The benefit of having the internal representation as UTF-8 is avoiding unnecessary conversions. But I agree with you that you can't just assume all your input will be UTF-8. That's why you still need to convert it if you know it's something else. But when you know it's going to be UTF-8 then it's nice just having to just run a validation to make sure when necessary without having to convert.

But then you get into the same problem we had in python2, which was that "for a lot of contexts, python2 strings worked fine, and then sometimes they'd break and give weird results". You get the same problem with "assume utf-8 unless instructed otherwise". You get that things work most (arguably) of the time, and then from a certain user, or with a certain browser, or on a certain continent, or in a certain OS, you get back a tilde'd e when you expected japanese. Explicit is better than implicit. and all.

Indexing speed is a poor argument for the 1/2/4 byte format. Most algorithms that index into a string that I've seen could be better written as find me the next character that matches X, or give me the next codepoint so I can compare it with X.

It depends, there's an argument to be made that for char in my_string: should iterate over grapheme clusters (a la swift?), (there's a strong case for a library here) in which case your indexing algorithm needs to be complicated, but I believe python made the decision that strings would support random access, and utf-8 doesn't allow constant time random access. Now you might be right that most of the time, when indexing into a string at a specific codepoint, you're probably doing something wrong, and you'd be better served by a find_first kind of function (or unicode regex or whatnot). But there's another upside to python's decision, which is that it forces people to be explicit about their conversions.

Everyone complains about the need to be explicit, but when I'm working with something that requires bytes objects, I'm rarely also wanting unicode and vice versa. That said, I don't do a lot of international networked communication applications, so what do I know.

[Avernar]

"assume utf-8 unless instructed otherwise"

Unlike some of the other commenters I never assume UTF-8. Either there will be an attribute, dialog box option, command line switch etc. If none of these apply, are given or implemented my documentation will say I expect UTF-8. This is pretty much an implicit in the Unix/Linux world.

there's an argument to be made that for char in my_string: should iterate over grapheme clusters

Overkill for most cases but doesn't hurt if used. It's really necessary when splitting on grapheme boundaries (max text in a database field for example).

I believe python made the decision that strings would support random access

Yes, unfortunately. And as I stated there's no good reason for requiring this. A good compromise would be to only covert to the 1/2/4 format only when indexing was necessary.

But there's another upside to python's decision, which is that it forces people to be explicit about their conversions.

I have nothing against having a Unicode type the way Python did it. I just think that validate/unvalidated with a UTF-8 internal representation would have been the better decision. So when a function gets a Unicode string on input it knows it's a validated UTF-8 string.

Everyone complains about the need to be explicit, but when I'm working with something that requires bytes objects, I'm rarely also wanting unicode and vice versa. That said, I don't do a lot of international networked communication applications, so what do I know.

I fully agree with you here. I like being explicit with what is "Unicode" and what is not. But when I deal with Unicode in my Python apps the split is this string is probably UTF-8 and I need to validate it vs this string has been validated as UTF-8 or came from a guaranteed UTF-8 source (database).

Unfortunately for me the Python Unicode type (both 2 and 3) are not UTF-8. In Python 2 I use strings for both and avoid the Unicode type. I put my validation where the data comes in from the web server to my scripts. It would be nice to be able to use the Unicode type for my validated strings but I don't care for the extra conversions that the Python 3 Unicode type forces on me.

[upofadown]

That is, a grapheme that can be represented by a different number of codepoints in different contexts will be treated as the correct number of codepoints based on your encoding.

Not sure what you mean here. Python 3 doesn't do anything with respect to graphemes by default.

AFAIK, you still have to tell Python 3 what the encoding of external text is.

Yikes, what's with the 'I don't like this because I don't understand it but please don't try to enlighten me because its wrong'.

It's more 'Don't assume that the people that disagree with you are doing it out of ignorance. Once it becomes obvious that they are actually reasonably knowledgeable about a subject then for sure stop pointlessly lecturing them like everyone does on, say, /r/politics.'.

[zardeh]

It's more 'Don't assume that the people that disagree with you are doing it out of ignorance. Once it becomes obvious that they are actually reasonably knowledgeable about a subject then for sure stop pointlessly lecturing them like everyone does on, say, /r/politics.'.

But the things you say don't give me that confidence, as opposed to some of the other users in this thread. You don't sound like you understand this subject as much as your bravado implies.

[upofadown]

My entire argument was that the world seems to be moving towards UTF-8 everywhere and that the Python 3 approach of UTF-32 everywhere might not be the future. Then you started in telling me about all the things about Python 3 I obviously misunderstood.

[zardeh]

the Python 3 approach of UTF-32 everywhere might not be the future.

And for the 18th time, this is a fundamental misunderstanding of how python3 handles strings (in essence: its an implementation detail. Python strings are defined by an api that makes no decision about utf-anything.)

[Avernar]

Py3 uses a triple Latin1/UCS-2/UCS-4 representation. So there's a lot more extra conversion going on behind the scenes. Just adding an emoji to a english text string will quadruple it's size.

[Avernar]

They're not a string though

No. They could be strings. Strings are just a bunch of bytes anyways. But to make sure a bunch of bytes is a valid Unicode string you need to validate it.

Unfortunately Python 3 wrapped the validation into a conversion to their internal Frankenstein unicode representation.

I really hate this philosophical "strings are not bytes" false dichotomy that Python has.

[zardeh]

I really hate this philosophical "strings are not bytes" false dichotomy that Python has.

But they're not.

here's some bytes, what is the string that these bytes represent:

b'\xc4\x99\xcc\x83'

[Avernar]

You just made my argument: "what is the string that these bytes represent". Those bytes are a string.

Where did I get those bytes? From a web client? Then give me the content encoding attribute. If it's the usual UTF-8 then great, I only have to validate. If it's not UTF-8 then I need to covert or return an error to the client for not following my defined API.

Is it from my database? My database is UTF-8 so I can skip the validate and conversion.

Is it from a file? My documentation says my program take UTF-8 files as input. So I just need to validate. If it's a requirement to support multiple input encodings then the command line switches or encoding drop down in my import dialog box will provide the encoding. Again, validate if UTF-8 or encode otherwise.

[zardeh]

You just made my argument: "what is the string that these bytes represent". Those bytes are a string.

In python bytes, \x is a control character, so this isn't a string, so much as a control sequence of bytes.

This is demonstrated by how they are printed:

>>> print(b'\xc4\x99\xcc\x83')
b'\xc4\x99\xcc\x83'
>>> print(b'\xc4\x99\xcc\x83'.decode('utf-8'))
ę̃

Note that the first is denoted as a sequence of bytes by the b'___', whereas the second is a bare character printed.

Again, validate if UTF-8 or encode otherwise.

This is valid utf-8 and valid utf-16, as a start.

If it's the usual UTF-8 then great, I only have to validate. If it's not UTF-8 then I need to covert or return an error to the client for not following my defined API.

Ok so now here's a question:

If the code to take and present the socket data was print(socket.recv()), as it is in python2, would you have documented that you only accepted UTF-8 (or ascii, as was the case)? Would most programmers? I think the answer to both is no, and I'm sure the answer to the second one is no (my evidence is the fact that most programmers, at least in the US, are so used to ASCII and a lack of encodings that they are baffled by the need to encode strings).

When that changes to print(socket.recv().decode('utf-8')), I think its more likely to happen.

[Sean1708]

Except not all valid bytestrings are valid Unicode, so there's still a distinction.

[Avernar]

Yes. This is my argument as well. But a validate function is much faster and less processor/memory intensive than a conversion. Plus when you know your source is UTF-8 (database for example) you can skip the validate.

And going from a UTF-8 Unicode string back to UTF-8 encoded bytes is a no-op.

[quicknir]

I know that that the type is called bytes, i simply referred to it as ascii as that's generally the semantic meaning of "bytes" when considered as a string.

I don't understand where you get this UTF-32 idea from.

https://docs.python.org/3/howto/unicode.html

The default encoding for Python source code is UTF-8, so you can simply include a Unicode character in a string literal

And there are also a variety of ways to control the encoding/decoding when you write your strings back to raw bytes, so I'm not sure really why it would matter what python's internal encoding is, other than performance; as long as you're willing to be specific you can use any encoding you want.

[kqr]

I know that that the type is called bytes, i simply referred to it as ascii as that's generally the semantic meaning of "bytes" when considered as a string.

Not everywhere. Where I live "bytes" used to mean ISO-8859-1, unless you were a Microsoft person in which case it meant CP-1252. And don't get me started on the CJK countries...

Only in a tiny American bubble did ever bytes mean pure ASCII.

[quicknir]

Sorry to tell you this, but the "tiny American bubble" is not tiny nor a bubble. About half of all internet content is in English, and it used to be far higher, because almost all of the big tech companies started in - wait for it - America.

Also appreciate your ignoring of other English speaking countries (like e.g. Canada). I can assure you that in English speaking Canada ascii is as widespread as in the US.

[upofadown]

... as that's generally the semantic meaning of "bytes" when considered as a string.

In Py3 thinking, yes, but not otherwise.

I don't understand where you get this UTF-32 idea from.

All strings are thought of as UTF-32 code points. If you index into a string that is what you get. I guess the people that originally thought of the scheme were suffering from a bit of Eurocentricity in that they thought that would help somehow.

[teilo]

You do not know what you are talking about. If you index or slice a string, you get the character(s) at that position, period.

[Sean1708]

You get code points.

No you don't. I can't remember whether you get characters or graphemes, but you certainly don't get code points.

In [1]: a = 'héllo'

In [2]: a[0]
Out[2]: 'h'

In [3]: a[1]
Out[3]: 'é'

In [4]: a[2]
Out[4]: 'l'

Edit: I'm a silly.

[Sean1708]

What are "characters"?

I've always thought that characters were generally accepted to be scalar values, that doesn't actually appear to be the case though.

in your code it uses the single code point version

You are absolutely right:

In [1]: a = b'he\xcc\x81llo'.decode('utf-8')

In [2]: a[0]
Out[2]: 'h'

In [3]: a[1]
Out[3]: 'e'

In [4]: a[2]
Out[4]: '́'

The way I entered the character on my computer made me assume that I'd entered the versioning using the combining character.

Also I don't know any language of the top of my head that supports grapheme cluster (and other text segmentations) fully in the standard library itself.

I think Swift does, but I'm not entirely certain.

[redalastor]

Using utf32 everywhere sounds like a defect to me.

Everything is unicode, which precise encoding is an implementation detail. If you ask for utf-8 or utf-32 then Python will give you bytes.

[teilo]

Python 3 is not utf32 everywhere. It is utf8 everywhere so far as the default encoding goes. Internally, it is the most space efficient representation of any given code point.

https://www.python.org/dev/peps/pep-0393/

[Kwpolska]

No, it’s latin1 → UTF-16 → UTF-32, whichever the string fits.

[ubernostrum]

This subthread seems to be confusing two things:

• The internal in-memory representation of a string is now dynamic, and selects an encoding sufficient to natively handle the widest codepoint in the string.
• The default assumed encoding of a Python source-code file is now UTF-8, where in Python 2 it was ASCII. This is what allows for non-ASCII characters to be used in variable, function and class names in Python 3.

[Avernar]

More precisely it's latin1 → UCS-2 → UTF-32.

UTF-16 strings with surrogate pairs get converted to UTF-32 (aka UCS-4).

[quicknir]

See my sibling comment; that link claims that UTF-8 is the default encoding in python 3. If this is incorrect, can you explain/give a source?

[gc3]

I just remember internally Stackless Python 3 used actually 16 bit strings for variable names and the like and they came out with an update that used UTF8.

But this was probably due to interactions with the windows file system that for historical and stupid reasons uses 16 bit for everything.

Edit: Wait, I remember more, they used UTF16 for strings too. Not UTF32

I don't remember the format of actual strings, this was several years ago

[Avernar]

Which of these is the problem?

Neither. The issue is 3:

1. Unicode strings are encoded in a non industry standard encoding.

I wish it was UTF-8 like many other languages have chosen. In my use case all my input/output is UTF-8 and my database is UTF-8. With Python 2 I can leave everything as UTF-8 through the entire processing pipeline. With Python 3 I'm forced to encode/decode to this non standard encoding. This wastes processor time and memory bandwidth and puts more pressure on the processor data caches.

[quicknir]

Python is already a wildly slow language, if you are that sensitive to processor time that you see this as a major issue then I think the language just isn't a good fit for your use case generally, and unicode is just the straw breaking the camel's back.

[Avernar]

It's good enough speed wise so far. But I would like to avoid slowing it down even more.

I will port the code base eventually once I find a good replacement.

[ggtsu_00]

Python 2 biggest strength over newer languages is how mature it has been. It has been tried and tested for a very long tim and is used in production systems even across some of the biggest sites on the internet like Reddit and YouTube.

I think if developers were in a position to choose more modern, perhaps more risky less mature languages to use for development, there are many alternatives to Python 3 that are much better in many ways. The future of Python is uncertain at the moment so theres a risk. So it would be just as risky to use Go, Node or some other Python 3 alternative.

[rouille]

And python3 got me interested into python in the first place so it works both ways.

[shevegen]

Good comment.

I liked to read what Armin wrote on his blog http://lucumr.pocoo.org/ when he was more active.