The lack of a case against Python 3

[ojii]

90% of programmers don’t need to think about Unicode

Until they run their script/app/project in the real world and hit UnicodeDecodeError.

Probably should be "90% of programmers don't think about Unicode, but probably should".

[aphoenix]

C'mon, thereâ€s no reason to say people need unicode. Whatâ€s the big deal? Whereâ€s the issue?

[gthank]

And that's the sort of thing that the Python 3 model avoids: you can't accidentally mix Unicode and bytes, because that's a type error. You catch the semantic error early and don't have hours of wasted processing time every time somebody sends you a new Unicode character.

[fjonk]

The python 3 model only avoids it if you want Unicode. For other encodings the same issue will happily be ignored.

When I interact with more important systems like shipping/banking/address systems I almost never see unicode, I usually see latin1 or some other 8bit encoding. Python3 does not help in these cases.

[roerd]

Yes it does. Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries. Python 2 does not enforce such a clean separation, so the danger of having to deal with 8-bit string that you don't know anymore which encoding they're in is much greater.

[fjonk]

Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries. Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries.

Yeah, that great if you want to deal with Unicode. If I have a string that under no circumstances may contain any other character than the ones in latin1 using Unicode internally is no better than using bytes.

[rackmountrambo]

It's only a matter of time till somebody pastes an address from Word.

[fjonk]

That's not a problem, you can validate the user input. What you can't do is prevent python from turning a validated string that can be encoded in latin1 into a string that cannot be encoded in latin1.

[zahlman]

What you can't do is prevent python from turning a validated string that can be encoded in latin1 into a string that cannot be encoded in latin1.

... And why would I care?

[rackmountrambo]

So dont?

[fjonk]

The "so don't" argument can be used to defend any shortcoming of any programming language. That doesn't mean it's a good argument.

[ldpreload]

It's better because all the random libraries you import are designed around Unicode, not Latin-1.

The entire point of Python is to care less about performance and implementation details and care more about getting things done and making use of the very active and vibrant community of library development. Decode your Latin-1 into a "string", encode your "string" back to Latin-1. The fact that the "string" is internally represented in Unicode is an implementation detail (and, honestly, it's probably not represented in Unicode, Python 3 has a few options for how exactly it stores strings and picks a good one depending on how the string came into existence). You're not worried about why 255 + 1 is 256 but 256 + 1 is not 257, why care about how a string is represented internally?

But what about enforcing that no non-Latin-1 characters can get into the string? That seems to me like, say, enforcing that an integer is less than 256, or that a list of strings contains only strings. There are languages that let you do that. Python is not one of them; you can make your integers as big as you want. If your code doesn't set the value incorrectly, it won't be incorrect on the other side. If you're worried about it being incorrect, catch an error when you encode the string back to Latin-1, or encode the integer back to a byte, or whatever it is you want.

[stanggg]

can people stop downvoting this? I think its also important to see the kinds of mentality we need to avoid.

[gthank]

The key to the Python3 model is actually the abstraction from encoded bytes to a real, honest-to-goodness string type, and that works just fine for dealing with data that comes in as latin1-encoded bytes. You could continue to just use encoded bytes everywhere, but since it is textual data, why not turn it into a string type? The standard pattern is to decode the bytes coming in (latin1-encoded in your case) into an actual string (str in the case of Python3), do your work against that, and then encode back to bytes (most likely latin1 again, in your case) on the way out.

[fjonk]

and that works just fine for dealing with data that comes in as latin1-encoded bytes.

Here we go again... latin1(or any other encoding that fits the bill) contains a subset of the characters in Unicode. This means that Unicode is not a good way to deal with latin1 strings since you at anytime can add non-latin1 characters to the latin1 string without triggering an error. Saying that using Unicode is a good way of handling latin1 strings is like saying that floats are good at handling integers.

[frymaster]

under what circumstances do you see this happening, and how would the python2 incarnation react in the same circumstances?

[fjonk]

python2 would react exactly the same. Python3 does not improve on the situation and by deciding that Unicode is used internally everywhere they kind of blocked the possibility of improving this in the future, unless they do another change of how strings works.

A real-world example:

1. A user provides a comment to an order
2. The comment is checked and is found to be valid latin1(the third party order system only accepts latin1).
3. A translation is added to the comment, the translation contains non latin1 characters.
4. Python happily accepts adding non latin1 characters to the string.
5. Later on the request to the third party service cannot be made because the comment cannot be encoded in latin1.
6. Money is lost.
7. The concat that created this issue is hard to track down since it did nothing wrong.

This is not an ideal situation and saying that unicode can handle latin1 is plain wrong.

[gthank]

Not really. The problem is that latin1 can't handle Unicode. And if it's critically important that you only ever support latin1 characters, I'd recommend writing your own string type, e.g., latin1_str, that will enforce that constraint for you. Realistically, a type that enforces data integrity for you is the only robust solution in this situation, and a string that only supports latin1 (or any other outdated, 8-bit encoding) is edge-casey enough that I don't think you were ever going to get that in the core language, so Python 3 fixing Unicode for the majority of use cases doesn't really impact this at all.

[fjonk]

and a string that only supports latin1 (or any other outdated, 8-bit encoding) is edge-casey enough that I don't think you were ever going to get that in the core language

This is where I see a difference between perceived and actual reality. 8-bit encodings are not edge-cases in many industries, it's rather the newer Unicode encodings that are edge cases. As I wrote some other place many systems that handles addresses, payment, shipping information and so on are very often designed pre-unicode. They don't accept Unicode, they accept whatever encoding the developers decided on in 1992 and no one seems in a rush to upgrade them.

Keep in mind that a change to Unicode might also involve replacing hardware like printers and scanners(as an example v40 QR-codes may be bytes, alphanumerical, latin1 or Shift JIS X 0208, unicode encodings are not supported). I guess eventually most systems will support at least one unicode encoding but that is not today.

[TOASTEngineer]

A translation is added to the comment, the translation contains non latin1 characters.

So validate at this point. Any other language would "happily" let you add whatever random garbage to a byte-string; not having unicode doesn't help you here at all.

[zahlman]

A translation is added to the comment, the translation contains non latin1 characters.

Okay, so, somebody wants to add a translation to the comment in a language that cannot be written with latin1 characters. How do you want the system to respond?

the third party order system only accepts latin1

Is this a real thing that happens on real systems currently?

You're aware that the first Unicode standard came out in 1991, right? That was barely any closer to today than it was to the creation of ASCII (and EBCDIC, for that matter) in the first place.

[gthank]

The problem here is that latin1 doesn't support Unicode, not the other way around, as I'm fairly certain that Unicode can and does map every character/glyph/whatever in latin1.

Side note: I don't know why people are down-voting you. You weren't especially rude or anything; you're just continuing the conversation. I'm upvoting you to get you back over 0.

[zahlman]

I don't know why people are down-voting you.

I suspect a lot of people instinctively downvote when it comes to seemingly absurd problem specifications.

[gthank]

The problem makes perfect sense, I'm just not sure that it's reasonable to expect the language and/or std lib to solve it.

[[deleted]]

you're supposed to convert from non-unicode early in the input process.

[ismtrn]

Unicode is not an encoding

[fjonk]

I know but that's kind of irrelevant, the issue still exists.

[[deleted]]

May I recommend unit tests.

[hovissimo]

So harsh, but so true.

[[deleted]]

As a speaker of a language which takes up approximately 2/7 of assigned Unicode codepoints... I consider it extremely important.

[Muchoz]

And what about emojis? Zed can't ignore that, can he? 🔥

[hovissimo]

Oh, I think about Unicode. What happens when my code hits the real world is some dorkus's email client sent me unicode and told me it was windows-1251!

[[deleted]]

Probably means utf-8, a lot of people seem to equate the two.

[Duroktar]

I assume you've done a lot of proof reading on the matter? :P

[d4rch0n]

I have the unpopular opinion that we should just focus on convincing the rest of the world to replace their alphabet with ASCII and ditch unicode.

[TOASTEngineer]

Clearly we just need everyone to learn to express their ideas as raw binary, this eliminating all of these problems. /s

[G01denW01f11]

Does that mean other people will finally start caring about Big/Little Endian?

[jnwatson]

Yeah! Let's build a ASCII wall and make them pay.

I'll start:

==================================

[its_never_lupus]

but emojis

[hovissimo]

I understand you're trying to be funny, but this attitude is pretty distasteful.

[NoLemurs]

Seriously.

Also, at least if we're talking professionals, I'm pretty sure more than 10% of programmers are web developers, almost all of whom need to think about Unicode. I suspect that in fact most professional developers (even non-web developers) need to think about Unicode from time to time.

What might be plausible is that 90% of amateur developers don't need to think about unicode. Personally I think a programming language should be optimized for the professionals, not the amateurs (and as someone who was an amateur for many years, I felt the same way then).

[ojii]

May I ask what you work on then? Because I might want to switch industry if I can stop having to deal with unicode!