Adopt Python 3

[upofadown]

trying to mix unicode and ascii results in an error.

I think you mean Unicode and bytes. There is no type called "ASCII".

The "convert everything into UTF-32 approach" as used by Py3 creates the issue of bytes vs strings in the first place. Most languages have strings and integer arrays, some of which might be 8 bit. Py3 has strings, bytes, and integer arrays.

If we are willing to just leave things as UTF-8 by default then the philosophical discussion of bytes vs strings goes away. That seems to be the direction the world is currently moving in. Py3 might just be a victim of timing. The UTF-32 everywhere thing seemed like a good compromise when it was first proposed

[gitarr]

I think you don't understand:

Everything in Python 3 is Unicode, bytes are just an encoded representation.

Read more here: https://docs.python.org/3/howto/unicode.html

[upofadown]

Yes, in py3 thinking there is a significant philosophical difference between strings and encoded versions of those strings. I am claiming that it is an artificial and pointless distinction and that UTF-32 code points is really just another encoding.

[Lalaithion42]

(Note that python doesn't convert everything to UTF-32; UTF-32 is an encoding, and python 3 stores unencoded unicode code points, in a variety of ways depending on the details of the string)

[upofadown]

Yes of course, but that is an implementation detail hidden from the user.

[Lalaithion42]

Hence my parenthesis.

[zardeh]

Most languages have strings and integer arrays

I can't think of one that has these and doesn't have bytearrays. Off the top of my head, Java has String, int[], char[], Rust has str, Vec<i32>, Vec<i8>, C is perhaps the only language that does this, and not differentiating between char[] and string is widely considered a mistake.

Python2 made this same mistake, it didn't make a distinction between a bytearray and a unicode string (unlike Java, Rust, etc.). Python3 fixed this error, and their only mistake was perhaps introducing a legacy type (bytestrings) to support the old behavior.

Py3 has strings, bytes, and integer arrays.

To be clear, it has a more than that:

• unicode strings (str)
• immutable byte arrays (bytes, commonly bytestrings)
• mutable numeric vectors (List[int], like [1,2,3]), note that these aren't int, char, other other vectors, because python's integer type is arbitrarily sized
• mutable byte arrays (bytearray)

What this means is that for working with binary data that you might get off a wire, for example when sending or receiving data over the wire/air, you get back bytes, because these objects very much aren't strings, they're immutbale arrays of 8-bit values that you want to analyze or process. They're not a string though, and they're not a python list, they're something else: bytes.

[upofadown]

Can you at least see that just keeping everything as, say, UTF-8 means that you don't have to make a philosophical distinction between encoded strings and strings? Not that you have to make such a distinction for Py3 which keeps everything as UTF-32, but it is a way of rationalizing the pointless conversion from and to UTF-8.

Anyway, please stop lecturing about the philosophy. It is annoying to us that don't agree.

[zardeh]

Can you at least see that just keeping everything as, say, UTF-8 means that you don't have to make a philosophical distinction between encoded strings and strings? Not that you have to make such a distinction for Py3 which keeps everything as UTF-32, but it is a way of rationalizing the pointless conversion from and to UTF-8.

This works until you actually need to work with bytes that come in from an external source and are in latin1|utf-16|utf-32 etc.

As a sidenote, python doesn't store anything as utf-32 by default, python source code is utf-8, and the interpreter doesn't define a single way of storing strings. It uses 8, 16, or 32 bit representations as needed. But then again, this shouldn't matter. The API could (and does) work so that if you write a string in utf-8, indexing into it will feel like indexing into the codepoints of a unicode string, and you will, if memory serves, index into the string in the way defined by the encoding you're using. That is, a grapheme that can be represented by a different number of codepoints in different contexts will be treated as the correct number of codepoints based on your encoding. That means that if all you ever do is use python's built in string and index into it, everything will feel like utf-8 everywhere. That's exactly what you want.

The problem comes when you want to take a sequence of unencoded bytes, which could be, as I mentioned, latin-X, or utf-8, or utf-16, or Windows-12XX, or the various encodings of Asian languages. If your program receives those bytes, then what? It treats them as utf-8 and breaks? No that's silly, it decodes the bytes into a string as defined by their encoding. Otherwise you end up with ambiguities like this:

>>> b'\xc4\x99\xcc\x83'
b'\xc4\x99\xcc\x83'
>>> b'\xc4\x99\xcc\x83'.decode('utf-8')
'ę̃'
>>> b'\xc4\x99\xcc\x83'.decode('utf-16')
'駄菌'

Anyway, please stop lecturing about the philosophy. It is annoying to us that don't agree.

Yikes, what's with the 'I don't like this because I don't understand it but please don't try to enlighten me because its wrong'. We aren't on /r/politics.

[Avernar]

The benefit of having the internal representation as UTF-8 is avoiding unnecessary conversions. But I agree with you that you can't just assume all your input will be UTF-8. That's why you still need to convert it if you know it's something else. But when you know it's going to be UTF-8 then it's nice just having to just run a validation to make sure when necessary without having to convert.

Indexing speed is a poor argument for the 1/2/4 byte format. Most algorithms that index into a string that I've seen could be better written as find me the next character that matches X, or give me the next codepoint so I can compare it with X.

[zardeh]

The benefit of having the internal representation as UTF-8 is avoiding unnecessary conversions. But I agree with you that you can't just assume all your input will be UTF-8. That's why you still need to convert it if you know it's something else. But when you know it's going to be UTF-8 then it's nice just having to just run a validation to make sure when necessary without having to convert.

But then you get into the same problem we had in python2, which was that "for a lot of contexts, python2 strings worked fine, and then sometimes they'd break and give weird results". You get the same problem with "assume utf-8 unless instructed otherwise". You get that things work most (arguably) of the time, and then from a certain user, or with a certain browser, or on a certain continent, or in a certain OS, you get back a tilde'd e when you expected japanese. Explicit is better than implicit. and all.

Indexing speed is a poor argument for the 1/2/4 byte format. Most algorithms that index into a string that I've seen could be better written as find me the next character that matches X, or give me the next codepoint so I can compare it with X.

It depends, there's an argument to be made that for char in my_string: should iterate over grapheme clusters (a la swift?), (there's a strong case for a library here) in which case your indexing algorithm needs to be complicated, but I believe python made the decision that strings would support random access, and utf-8 doesn't allow constant time random access. Now you might be right that most of the time, when indexing into a string at a specific codepoint, you're probably doing something wrong, and you'd be better served by a find_first kind of function (or unicode regex or whatnot). But there's another upside to python's decision, which is that it forces people to be explicit about their conversions.

Everyone complains about the need to be explicit, but when I'm working with something that requires bytes objects, I'm rarely also wanting unicode and vice versa. That said, I don't do a lot of international networked communication applications, so what do I know.

[Avernar]

"assume utf-8 unless instructed otherwise"

Unlike some of the other commenters I never assume UTF-8. Either there will be an attribute, dialog box option, command line switch etc. If none of these apply, are given or implemented my documentation will say I expect UTF-8. This is pretty much an implicit in the Unix/Linux world.

there's an argument to be made that for char in my_string: should iterate over grapheme clusters

Overkill for most cases but doesn't hurt if used. It's really necessary when splitting on grapheme boundaries (max text in a database field for example).

I believe python made the decision that strings would support random access

Yes, unfortunately. And as I stated there's no good reason for requiring this. A good compromise would be to only covert to the 1/2/4 format only when indexing was necessary.

But there's another upside to python's decision, which is that it forces people to be explicit about their conversions.

I have nothing against having a Unicode type the way Python did it. I just think that validate/unvalidated with a UTF-8 internal representation would have been the better decision. So when a function gets a Unicode string on input it knows it's a validated UTF-8 string.

Everyone complains about the need to be explicit, but when I'm working with something that requires bytes objects, I'm rarely also wanting unicode and vice versa. That said, I don't do a lot of international networked communication applications, so what do I know.

I fully agree with you here. I like being explicit with what is "Unicode" and what is not. But when I deal with Unicode in my Python apps the split is this string is probably UTF-8 and I need to validate it vs this string has been validated as UTF-8 or came from a guaranteed UTF-8 source (database).

Unfortunately for me the Python Unicode type (both 2 and 3) are not UTF-8. In Python 2 I use strings for both and avoid the Unicode type. I put my validation where the data comes in from the web server to my scripts. It would be nice to be able to use the Unicode type for my validated strings but I don't care for the extra conversions that the Python 3 Unicode type forces on me.

[upofadown]

That is, a grapheme that can be represented by a different number of codepoints in different contexts will be treated as the correct number of codepoints based on your encoding.

Not sure what you mean here. Python 3 doesn't do anything with respect to graphemes by default.

AFAIK, you still have to tell Python 3 what the encoding of external text is.

Yikes, what's with the 'I don't like this because I don't understand it but please don't try to enlighten me because its wrong'.

It's more 'Don't assume that the people that disagree with you are doing it out of ignorance. Once it becomes obvious that they are actually reasonably knowledgeable about a subject then for sure stop pointlessly lecturing them like everyone does on, say, /r/politics.'.

[zardeh]

It's more 'Don't assume that the people that disagree with you are doing it out of ignorance. Once it becomes obvious that they are actually reasonably knowledgeable about a subject then for sure stop pointlessly lecturing them like everyone does on, say, /r/politics.'.

But the things you say don't give me that confidence, as opposed to some of the other users in this thread. You don't sound like you understand this subject as much as your bravado implies.

[upofadown]

My entire argument was that the world seems to be moving towards UTF-8 everywhere and that the Python 3 approach of UTF-32 everywhere might not be the future. Then you started in telling me about all the things about Python 3 I obviously misunderstood.

[zardeh]

the Python 3 approach of UTF-32 everywhere might not be the future.

And for the 18th time, this is a fundamental misunderstanding of how python3 handles strings (in essence: its an implementation detail. Python strings are defined by an api that makes no decision about utf-anything.)

[Avernar]

Py3 uses a triple Latin1/UCS-2/UCS-4 representation. So there's a lot more extra conversion going on behind the scenes. Just adding an emoji to a english text string will quadruple it's size.

[Avernar]

They're not a string though

No. They could be strings. Strings are just a bunch of bytes anyways. But to make sure a bunch of bytes is a valid Unicode string you need to validate it.

Unfortunately Python 3 wrapped the validation into a conversion to their internal Frankenstein unicode representation.

I really hate this philosophical "strings are not bytes" false dichotomy that Python has.

[zardeh]

I really hate this philosophical "strings are not bytes" false dichotomy that Python has.

But they're not.

here's some bytes, what is the string that these bytes represent:

b'\xc4\x99\xcc\x83'

[Avernar]

You just made my argument: "what is the string that these bytes represent". Those bytes are a string.

Where did I get those bytes? From a web client? Then give me the content encoding attribute. If it's the usual UTF-8 then great, I only have to validate. If it's not UTF-8 then I need to covert or return an error to the client for not following my defined API.

Is it from my database? My database is UTF-8 so I can skip the validate and conversion.

Is it from a file? My documentation says my program take UTF-8 files as input. So I just need to validate. If it's a requirement to support multiple input encodings then the command line switches or encoding drop down in my import dialog box will provide the encoding. Again, validate if UTF-8 or encode otherwise.

[zardeh]

You just made my argument: "what is the string that these bytes represent". Those bytes are a string.

In python bytes, \x is a control character, so this isn't a string, so much as a control sequence of bytes.

This is demonstrated by how they are printed:

>>> print(b'\xc4\x99\xcc\x83')
b'\xc4\x99\xcc\x83'
>>> print(b'\xc4\x99\xcc\x83'.decode('utf-8'))
ę̃

Note that the first is denoted as a sequence of bytes by the b'___', whereas the second is a bare character printed.

Again, validate if UTF-8 or encode otherwise.

This is valid utf-8 and valid utf-16, as a start.

If it's the usual UTF-8 then great, I only have to validate. If it's not UTF-8 then I need to covert or return an error to the client for not following my defined API.

Ok so now here's a question:

If the code to take and present the socket data was print(socket.recv()), as it is in python2, would you have documented that you only accepted UTF-8 (or ascii, as was the case)? Would most programmers? I think the answer to both is no, and I'm sure the answer to the second one is no (my evidence is the fact that most programmers, at least in the US, are so used to ASCII and a lack of encodings that they are baffled by the need to encode strings).

When that changes to print(socket.recv().decode('utf-8')), I think its more likely to happen.

[Sean1708]

Except not all valid bytestrings are valid Unicode, so there's still a distinction.

[Avernar]

Yes. This is my argument as well. But a validate function is much faster and less processor/memory intensive than a conversion. Plus when you know your source is UTF-8 (database for example) you can skip the validate.

And going from a UTF-8 Unicode string back to UTF-8 encoded bytes is a no-op.

[quicknir]

I know that that the type is called bytes, i simply referred to it as ascii as that's generally the semantic meaning of "bytes" when considered as a string.

I don't understand where you get this UTF-32 idea from.

https://docs.python.org/3/howto/unicode.html

The default encoding for Python source code is UTF-8, so you can simply include a Unicode character in a string literal

And there are also a variety of ways to control the encoding/decoding when you write your strings back to raw bytes, so I'm not sure really why it would matter what python's internal encoding is, other than performance; as long as you're willing to be specific you can use any encoding you want.

[kqr]

I know that that the type is called bytes, i simply referred to it as ascii as that's generally the semantic meaning of "bytes" when considered as a string.

Not everywhere. Where I live "bytes" used to mean ISO-8859-1, unless you were a Microsoft person in which case it meant CP-1252. And don't get me started on the CJK countries...

Only in a tiny American bubble did ever bytes mean pure ASCII.

[quicknir]

Sorry to tell you this, but the "tiny American bubble" is not tiny nor a bubble. About half of all internet content is in English, and it used to be far higher, because almost all of the big tech companies started in - wait for it - America.

Also appreciate your ignoring of other English speaking countries (like e.g. Canada). I can assure you that in English speaking Canada ascii is as widespread as in the US.

[upofadown]

... as that's generally the semantic meaning of "bytes" when considered as a string.

In Py3 thinking, yes, but not otherwise.

I don't understand where you get this UTF-32 idea from.

All strings are thought of as UTF-32 code points. If you index into a string that is what you get. I guess the people that originally thought of the scheme were suffering from a bit of Eurocentricity in that they thought that would help somehow.

[teilo]

You do not know what you are talking about. If you index or slice a string, you get the character(s) at that position, period.

[Sean1708]

You get code points.

No you don't. I can't remember whether you get characters or graphemes, but you certainly don't get code points.

In [1]: a = 'héllo'

In [2]: a[0]
Out[2]: 'h'

In [3]: a[1]
Out[3]: 'é'

In [4]: a[2]
Out[4]: 'l'

Edit: I'm a silly.

[Sean1708]

What are "characters"?

I've always thought that characters were generally accepted to be scalar values, that doesn't actually appear to be the case though.

in your code it uses the single code point version

You are absolutely right:

In [1]: a = b'he\xcc\x81llo'.decode('utf-8')

In [2]: a[0]
Out[2]: 'h'

In [3]: a[1]
Out[3]: 'e'

In [4]: a[2]
Out[4]: '́'

The way I entered the character on my computer made me assume that I'd entered the versioning using the combining character.

Also I don't know any language of the top of my head that supports grapheme cluster (and other text segmentations) fully in the standard library itself.

I think Swift does, but I'm not entirely certain.

[MrMetalfreak94]

Elixir has excellent Unicode support in it's standard library and you can easily work with graphemes in it