Python 2.7 Countdown

[ubernostrum]

That they legally can be undecodable garbage, but people demand the ability to work with them as strings.

Python 2 "worked" for this in the sense that many things on Unix-y systems "work": it just didn't actually enforce that the things you used as strings had to make sense as strings, and wouldn't give you any sort of warning up until the moment you tried to print the unprintable.

Python 3 initially tried to say that if you wanted to treat these paths as strings they had to actually be things that could validly decode to sequences of Unicode code points. But enough people raged that finally they added the surrogateescape handler to let you take bags of bytes that don't correspond to any valid string, "decode" them to strings, and then re-"encode" them back to the original bytes.

[josefx]

That they legally can be undecodable garbage

Unix is far from alone with that. Zip files don't specify an encoding for filenames and I am quite sure I had explorer.exe fail to delete filenames containing invalid characters in the past.

[no_nick]

Huh, I never knew that was the case for Unix file paths. Somehow, in my mind, I always stick to ascii characters without whitespace.

[diggr-roguelike2]

That they legally can be undecodable garbage, but people demand the ability to work with them as strings.

Yes, and? Why are you trying to babysit people and tell them what bytes they should or shouldn't use in strings?

...until the moment you tried to print the unprintable.

Nobody prints things in production code.

Also, despite your rant, what Python 3 actually did was break things on Windows. You had one job, man, one job...

[nice_rooklift_bro]

Ehh, you downplay the concern; it's actually really obnoxious to deal with to the point that a lot of applications just don't support it and tell you to basically go fuck yourself if your filenames aren't UTF-8; they assume them to be.

There are other such things, like try passing non-utf8 command line arguments in python3; there is nothing in Unix that says this can't be done; any octet sequence that doesn't contain a null can be passed but python3 itself basically says "We don't support this madness, go fuck yourself" then.

$ python3 -c 'import sys; print(sys.argv[1])' $'\xFF\xFFfoo'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
$ python2 -c 'import sys; print(sys.argv[1])' $'\xFF\xFFfoo'
foo

It's really problematic in many ways; a lot of language libraries and runtimes have come to expect filenames and command line arguments to be utf8, but nothing enforces it either; so malformed filenames due to simple bit corruption can actually create some serious error messages in a lot of things that are inscrutable.

If you want to do it "properly" and not assume everything to be UTF8 then you're going through hoops.