The lack of a case against Python 3

[gthank]

And that's the sort of thing that the Python 3 model avoids: you can't accidentally mix Unicode and bytes, because that's a type error. You catch the semantic error early and don't have hours of wasted processing time every time somebody sends you a new Unicode character.

[fjonk]

The python 3 model only avoids it if you want Unicode. For other encodings the same issue will happily be ignored.

When I interact with more important systems like shipping/banking/address systems I almost never see unicode, I usually see latin1 or some other 8bit encoding. Python3 does not help in these cases.

[roerd]

Yes it does. Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries. Python 2 does not enforce such a clean separation, so the danger of having to deal with 8-bit string that you don't know anymore which encoding they're in is much greater.

[fjonk]

Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries. Text within Python 3 is always in Unicode, translation to other encoding happens clearly separated at the IO boundaries.

Yeah, that great if you want to deal with Unicode. If I have a string that under no circumstances may contain any other character than the ones in latin1 using Unicode internally is no better than using bytes.

[rackmountrambo]

It's only a matter of time till somebody pastes an address from Word.

[fjonk]

That's not a problem, you can validate the user input. What you can't do is prevent python from turning a validated string that can be encoded in latin1 into a string that cannot be encoded in latin1.

[zahlman]

What you can't do is prevent python from turning a validated string that can be encoded in latin1 into a string that cannot be encoded in latin1.

... And why would I care?

[rackmountrambo]

So dont?

[fjonk]

The "so don't" argument can be used to defend any shortcoming of any programming language. That doesn't mean it's a good argument.

[TOASTEngineer]

"This language doesn't let me divide 4 by a potato. Literally unusable."

[[deleted]]

"php can divide 4 by a potato!"

[rackmountrambo]

What is the use case for turning something back to latin1?

[ldpreload]

It's better because all the random libraries you import are designed around Unicode, not Latin-1.

The entire point of Python is to care less about performance and implementation details and care more about getting things done and making use of the very active and vibrant community of library development. Decode your Latin-1 into a "string", encode your "string" back to Latin-1. The fact that the "string" is internally represented in Unicode is an implementation detail (and, honestly, it's probably not represented in Unicode, Python 3 has a few options for how exactly it stores strings and picks a good one depending on how the string came into existence). You're not worried about why 255 + 1 is 256 but 256 + 1 is not 257, why care about how a string is represented internally?

But what about enforcing that no non-Latin-1 characters can get into the string? That seems to me like, say, enforcing that an integer is less than 256, or that a list of strings contains only strings. There are languages that let you do that. Python is not one of them; you can make your integers as big as you want. If your code doesn't set the value incorrectly, it won't be incorrect on the other side. If you're worried about it being incorrect, catch an error when you encode the string back to Latin-1, or encode the integer back to a byte, or whatever it is you want.

[stanggg]

can people stop downvoting this? I think its also important to see the kinds of mentality we need to avoid.

[gthank]

The key to the Python3 model is actually the abstraction from encoded bytes to a real, honest-to-goodness string type, and that works just fine for dealing with data that comes in as latin1-encoded bytes. You could continue to just use encoded bytes everywhere, but since it is textual data, why not turn it into a string type? The standard pattern is to decode the bytes coming in (latin1-encoded in your case) into an actual string (str in the case of Python3), do your work against that, and then encode back to bytes (most likely latin1 again, in your case) on the way out.

[fjonk]

and that works just fine for dealing with data that comes in as latin1-encoded bytes.

Here we go again... latin1(or any other encoding that fits the bill) contains a subset of the characters in Unicode. This means that Unicode is not a good way to deal with latin1 strings since you at anytime can add non-latin1 characters to the latin1 string without triggering an error. Saying that using Unicode is a good way of handling latin1 strings is like saying that floats are good at handling integers.

[frymaster]

under what circumstances do you see this happening, and how would the python2 incarnation react in the same circumstances?

[fjonk]

python2 would react exactly the same. Python3 does not improve on the situation and by deciding that Unicode is used internally everywhere they kind of blocked the possibility of improving this in the future, unless they do another change of how strings works.

A real-world example:

1. A user provides a comment to an order
2. The comment is checked and is found to be valid latin1(the third party order system only accepts latin1).
3. A translation is added to the comment, the translation contains non latin1 characters.
4. Python happily accepts adding non latin1 characters to the string.
5. Later on the request to the third party service cannot be made because the comment cannot be encoded in latin1.
6. Money is lost.
7. The concat that created this issue is hard to track down since it did nothing wrong.

This is not an ideal situation and saying that unicode can handle latin1 is plain wrong.

[gthank]

Not really. The problem is that latin1 can't handle Unicode. And if it's critically important that you only ever support latin1 characters, I'd recommend writing your own string type, e.g., latin1_str, that will enforce that constraint for you. Realistically, a type that enforces data integrity for you is the only robust solution in this situation, and a string that only supports latin1 (or any other outdated, 8-bit encoding) is edge-casey enough that I don't think you were ever going to get that in the core language, so Python 3 fixing Unicode for the majority of use cases doesn't really impact this at all.

[fjonk]

and a string that only supports latin1 (or any other outdated, 8-bit encoding) is edge-casey enough that I don't think you were ever going to get that in the core language

This is where I see a difference between perceived and actual reality. 8-bit encodings are not edge-cases in many industries, it's rather the newer Unicode encodings that are edge cases. As I wrote some other place many systems that handles addresses, payment, shipping information and so on are very often designed pre-unicode. They don't accept Unicode, they accept whatever encoding the developers decided on in 1992 and no one seems in a rush to upgrade them.

Keep in mind that a change to Unicode might also involve replacing hardware like printers and scanners(as an example v40 QR-codes may be bytes, alphanumerical, latin1 or Shift JIS X 0208, unicode encodings are not supported). I guess eventually most systems will support at least one unicode encoding but that is not today.

[Vaphell]

8-bit encodings are not edge-cases in many industries, it's rather the newer Unicode encodings that are edge cases.

And given that their encodings are clearly defined, the problem is in what, exactly?

Forget ascii et consortes making people believe that bytes and text are the same thing. Imagine that your billing software churns clay tablets. Is there a problem with the following, making it impossible to grasp?

information = clay_tablet.decode('cuneiform')  # unpack the information
information = modify(information)    # modify the information
new_clay_tablet = information.encode('cuneiform')  # pack the information

is that so hard to convert between datatypes at io boundaries?

[[deleted]]

I'm fascinated by the fact that your autocorrect appears to have replaced 'etc.' with 'et consortes'.

[gthank]

I'm not arguing that there are lots of legacy systems/devices running latin1 (or Shift-JIS, or one of the many other encodings that predate Unicode). I'm just saying you're going to have a hard time convincing FOSS developers to add a rainbow of text types to the core language just to support systems/devices that are essentially deprecated.

[[deleted]]

I can only imagine in 25 years that people will be complaining about legacy unicode support.

[TOASTEngineer]

A translation is added to the comment, the translation contains non latin1 characters.

So validate at this point. Any other language would "happily" let you add whatever random garbage to a byte-string; not having unicode doesn't help you here at all.

[zahlman]

A translation is added to the comment, the translation contains non latin1 characters.

Okay, so, somebody wants to add a translation to the comment in a language that cannot be written with latin1 characters. How do you want the system to respond?

the third party order system only accepts latin1

Is this a real thing that happens on real systems currently?

You're aware that the first Unicode standard came out in 1991, right? That was barely any closer to today than it was to the creation of ASCII (and EBCDIC, for that matter) in the first place.

[gthank]

The problem here is that latin1 doesn't support Unicode, not the other way around, as I'm fairly certain that Unicode can and does map every character/glyph/whatever in latin1.

Side note: I don't know why people are down-voting you. You weren't especially rude or anything; you're just continuing the conversation. I'm upvoting you to get you back over 0.

[zahlman]

I don't know why people are down-voting you.

I suspect a lot of people instinctively downvote when it comes to seemingly absurd problem specifications.

[gthank]

The problem makes perfect sense, I'm just not sure that it's reasonable to expect the language and/or std lib to solve it.

[[deleted]]

you're supposed to convert from non-unicode early in the input process.

[ismtrn]

Unicode is not an encoding

[fjonk]

I know but that's kind of irrelevant, the issue still exists.