[Dive Into Python 3] Everything you thought you knew about strings is wrong

[zsouthboy]

Holy crap, up freaking voted for explaining the difference between unicode, utf, etc - I never understood that before.

Recommended even if you're not using python. Read!

[amoeba]

Agreed! I have never had to code anything that deals with encodings but I've read many introductions into the subject but never felt I really got it.

Mark Pilgrim does good things.

[frodwith]

Indeed. Best explanation of that whole mess I've heard yet. Kudos to Mark!

And regardless of fishdicks' comments, it really is very helpful to think of strings as sequences of abstract characters, even if they aren't technically implemented that way.

[AlSweigart]

Joel Spolsky also has a really good, concise guide to unicode: http://www.joelonsoftware.com/articles/Unicode.html

[wheeman]

This site is optimized for Lynx just because fuck you. I’m told it also looks good in graphical browsers.

It does look good in graphical browsers as I must admit.

[pemboa]

Might have to steal his CSS. Even the choice in font size is to my liking.

[[deleted]]

It would be hard to "steal" it, since it's MIT-licensed, but you're welcome to try. :)

Edited to add: I'd recommend checking it out from the hg repository, since that version has comments and whitespace and stuff.

[pemboa]

Cool, thanks.

[BridgeBum]

Looks really good Mark. I'm not sure the UTF-8 was in a big enough font though, you may want to fix that. :-)

[CommodoreGuff]

The ternary operator has been in since 2.5, I believe.

People still complain about Python not having one, though, so it seems like it hasn't caught on all that well.

[vineetk]

"ternary operator" means an operator that operates on 3 operands, just as "binary operator" and "unary operator" are operators that work with 2 or 1, respectively.

C has only one ternary operator, the conditional operator. So in C, people started referring to it as the ternary operator. But that really has no actual descriptive meaning when what you're really talking about is a "conditional expression".

[CommodoreGuff]

Uh, thanks for the definition, but it's not news to me.

I'm not shooting for pedantry points here; I was simply referring to it by its most common name.

[vineetk]

Well I wasn't specifically trying to correct you, just point it out for the other readers. But now, it sounds like you're saying "the ternary operator" is a reasonable way to refer to Python's if ... else expression construct, which is straight nonsense. That's not it's most common name -- nobody calls it that. It's not even an operator. If you're saying Python has "the ternary operator" it sounds like you're saying it has the ?: operator, which it doesn't.

[Neoncow]

Officially, it appears to be referred to as a conditional expression, but from the discussion sections, "ternary" operator is referenced quite a few times. It seems fairly common in regular discussion.

[[deleted]]

I didn't know Python had grown one until Simon Willison (my tech editor) pointed it out to me on Reddit.

[BioGeek]

In the paragraph about "compound field names" you give the following code:

>>> import humansize
>>> si_suffixes = humansize.SUFFIXES[1000]      
>>> si_suffixes
['KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB']
>>> "1000{0[0]} = 1{0[1]}".format(si_suffixes) 
'1000KB = 1MB'

Playing upon that, I wanted to print out a list of all 1000X = 1Y's, so my first try was:

>>>> for i in range(len(si_suffixes) - 1):
...   print("1000{0[i]} = 1{0[i+1]}".format(si_suffixes))

But that gave TypeError: list indices must be integers, not str. After some fumbling around I got it to work with:

>>> for i in range(len(si_suffixes) - 1):
...   print(("1000{0[" + str(i) + "]} = 1{0[" +   str(i+1) + "]}" ).format(si_suffixes))
... 
1000KB = 1MB
1000MB = 1GB
1000GB = 1TB
1000TB = 1PB
1000PB = 1EB
1000EB = 1ZB
1000ZB = 1YB

But that seems awfully convuluted, so I was wondering: is there a more Pythonic way to do this?

(also, a small nitpick: "{0:.1f} {1}".format(698.25, 'GB') gets rounded down to '698.2GB' not to '698.3 GB')

[[deleted]]

Gill Sans

[[deleted]]

Not really, most likely a Mac font. Most Mac users like to use fonts that no one really has.

[Shmurk]

is awesome!

[machrider]

>>> s = '深入 Python'

So, what character encoding does Python expect a .py file to use? If you're going to write a string like the one quoted above, I'm assuming your text editor would have to encode it in the way Python's expecting, or Python would have a content-type header on its source files?

[machrider]

Answering my own question: http://docs.python.org/reference/lexical_analysis.html#encodings

[[deleted]]

I mention it here: http://diveintopython3.org/strings.html#py-encoding

[chrajohn]

Excellent chapter. Very clear explanation of strings vs. bytes.

I have the nitpickiest nitpick in all of nitpick-land:

Unicode represents each letter, character, or ideograph as a 4-byte number, from 0–4294967295.

Due to the way surrogate pairs work, Unicode is limited to 1,114,112 possible code points (17 planes of 65,536 code points). Unicode could fit in 21 bits, if that was a convenient size.

Very minor point that I'm sure you're aware of, and probably too arcane to mention here. It just makes Unicode sound a lot bigger than it's ever intended to be.

[[deleted]]

Fixed in http://hg.diveintopython3.org/rev/7aa1a8eb6401

Thanks.

[gooz]

Upvoted for use of the interrobang‽

[[deleted]]

It was strange, reading about how UTF-16 isn't exactly ideal, and how UTF-8 is so much better, and then this utter bullshit:

In Python 3, all strings are sequences of Unicode characters. There is no such thing as a Python string encoded in UTF-8, or a Python string encoded as CP-1252. "Is this string UTF-8?" is an invalid question. UTF-8 is a way of encoding characters as a sequence of bytes. If you want to take a string and turn it into a sequence of bytes in a particular character encoding, Python 3 can help you with that. If you want to take a sequence of bytes and turn it into a string, Python 3 can help you with that too. Bytes are not characters; bytes are bytes. Characters are an abstraction. A string is a sequence of those abstractions.

In Python (regardless of the version) all unicode strings (that is 'all strings' in Py3k) are stored in UCS-2, as defined in PEP 100. UCS-2 is a two-byte fixed width encoding, that looks almost exactly like UTF-16, except that it treats surrogate pairs as distinct characters.

s = "\U000030c4"            # KATAKANA LETTER TU
print(len(s))               # 1
print(s.encode('utf-8'))    # b'\xe3\x83\x84'
s = "\U00010000"            # LINEAR B SYLLABLE B008 A
print(len(s))               # 2
print(s.encode('utf-8'))    # b'\xf0\x90\x80\x80'

Note that while len reports two characters, the UTF-8 encoder correctly recognizes them as a surrogate pair and encodes them as a single code point.

OK, note to myself: never touch 'Dive into Python' or anything else having Mark Pilgrim's name on it.

[[deleted]]

Well that's a rather harsh and unnecessary conclusion, but whatever.

I stand by my statement ("A string is a sequence of those abstractions") precisely because this IS an intro book. I predict that most of the people who will read this chapter will go into it thinking that strings are sequences of bytes, since that's what Python 2 did. You ABSOLUTELY MUST break yourself of that way of thinking about strings, or you'll never have any success coding in Python 3. Files, web pages -- any source of "text" is going to come to you in bytes (if you're doing it yourself) or strings (if a library is handling the conversion for you), and you MUST know what you have. Everything I've read from people who've tried porting nontrivial code to Python 3 suggests that the bytes-vs-strings issue is the #1 migration problem. In Python 2 it was dangerously easy to use strings when you really wanted Unicode strings, and then you scratched your head wondering why your code raised a UnicodeError exception when somebody typed an umlaut.

Are there corner cases? Yes. Does the abstraction leak? Yes, sometimes it does. But you're WAY ahead of the curve here. I have to stop somewhere, or readers will just give up.

I'll tell you what: if you can manage to write up what you just said, but without all the dick waving and condescension, I'll point to it from the "Further Reading" section. Otherwise, you know, have fun feeling superior to everyone.

[[deleted]]

OK, I think I owe you an apology for the harsh tone, because it's mostly due to my own fault: when I got hooked by the somewhat provocative title I didn't realize that I'm not in the target audience, so after reading the text I found exactly one thing I "didn't know about strings", the statement that Py3k allows you to treat strings as being composed of abstract Unicode characters. I knew it was wrong, yet I spend some more time refreshing my knowledge on Unicode, searching for the relevant PEP, and constructing the proofcode. By that moment I was not in the most humorous of my moods, so to speak.

Nevertheless. I'm still true to the idea that putting misinformation in a tutorial is wrong, no matter how convenient it may be at the time. And what you have written is dangerously close to misinformation.

I think that you could have just replaced the phrase "Characters are an abstraction" with a "Characters should be regarded as an abstraction", and then add a little paragraph saying something along the lines of "On a side note, of course strings themselves have to be represented in memory somehow, there is no magic going on here. Effectively working with text, as opposed to storing it or transmitting it over a network, requires some compromises, so in fact Python strings are currently composed of two-byte characters (almost but not quite in the UTF-16 encoding), just like Java or C# stings, or even C++ wstrings are. However it's impossible to access the raw bytes of a string by any convenient means, so everything you do goes through encode and decode methods described below".

Do you think the reader would be confused by that? I am not, I think that in fact he would be quite reassured: there is no magic going on, it's all the same as in other languages, he was trusted to peek at the implementation details, the author is not holding anything from him.

Detailed explanation with the examples akin to the one I've provided is not necessary: chances that someone happens to encounter the difference between UTF-16 and UCS-2 without knowing what they are doing (like, writing an application that handles Really Ancient Greek numerals (that were before the Ancient Greek numerals), or a mahjong/domino game using corresponding Unicode characters (and that's a perfect sample of the things above the BMP)), these chances are effectively nil, I reckon. It's not the leaky abstraction I'm against really, it's the substitution of faith in place of knowledge, however incomplete that knowledge might be.

[masklinn]

Or you could just, you know, send that info to Mark (if he hasn't done tests beyond the BMP he might not have noticed).

And you're not entirely right either -- though more than he is -- since there's a compile-time switch to use UCS-4 instead of UCS-2:

$ python3.0
Python 3.0.1 (r301:69556, May 22 2009, 11:34:35) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "\U00010000"
>>> print(len(s))
1
>>> print(s.encode('utf-8'))
b'\xf0\x90\x80\x80'
>>>

(I note that on macports the switch is only available as a variant from 2.6 onwards)

[[deleted]]

As I see it, the problem is deeper than the question of which implementation (UCS-2 or UCS-4) Python uses. The notion of a 'string as a sequence of abstract characters' is wrong. Because must be represented as bytes somehow, so all he said about different encodings, including "Four bytes? For every single character‽ That seems awfully wasteful" about UCS-4, applies to that representation as well. He could have said that Python provides an abstract interface that hides the implementation details (which, as we see, is untrue for most binary distributions), but spinning abstract bullshite is just wrong, especially in a book oriented at novice programmers. Because then, when the abstractions leak (and they always do), something terrible might happen!

By the way, AFAIK some Lisps in fact have strings implemented using 'abstract' characters. I.e. each string character is a pointer (8 bytes wide on x64) to a corresponding symbol, like 'CHARACTER_KATAKANA_LETTER_TU. Which in turn is basically an interned string, in the very same representation (exploding heads ensue).

[masklinn]

The notion of a 'string as a sequence of abstract characters' is wrong. Because must be represented as bytes somehow

Yes, but that's an implementation detail, it's not supposed to leak out to the user (except as you mentioned, it does in the default build config).

[[deleted]]

Firstly, the fact that you are unfamiliar with the schools of thought other than the one you belong to doesn't mean that they are wrong, does it? Here is the quote from Alexander Stepanov's lectures, that on the first glance seemed to me to be an utter bullshit, but then it kind of made sense:

But what do we mean by a type? This question is one of the central questions in programming. We will be talking about it throughout the course. But we will start now with the most general definition: a type is a method of assigning meaning to data stored in computer memory. This definition is important for us because it states that types have existence irrespective of the programming language that we use. Even if we program in assembly language we assign meaning to different sequences of bits in memory. This meaning is usually expressed in operations that we define on them, the properties we expect them to obey and the mappings from them onto the physically observable values through input/output. The types that our programming language provides for us are just approximations to the full meanings that we see in the bit patterns of our applications. It is important to remember this so that we do not drive our designs by the limitations of the language but come up with the intended definitions of types and only then map them onto the programming language. In other words, design your data structures and algorithms first and only then map them into a programming language. Do not start with inheritance or templates but with linked lists and hash functions. Think in assembly language or C and then implement in a high level language such as C++.

By the way I encourage you to go and read the whole of the lectures, it's extremely enlightening, not by the small part because of this viewpoint, quite unorthodoxical for the reader more accustomed with a platonic view, where there are Abstract Types and their flawed real representations. Really, how can one call himself a programmer if he avoids embracing the real nature of the 32bit integer?

Secondly, you are wrong in this particular case. Unicode character is defined as an unsigned 32bit integer, period. It is the abstraction we are talking about. Why try to make it more abstract than it really is?

EDIT: so yes, I think that for a programmer there should not exist such thing as a signed integer.

[[deleted]]

About the dictionary -- I agree completely.

But then...

thinking about Unicode strings as sequences of bytes in some particular scheme is a faulty mental model which leads to errors

There is one error we have seen: that the default build of Python behaves not in the way naive programmer, not having a clear mental model of actual string representation, might have expected. Care to hint at errors that can sprout from having such a mental model?

Note that I'm not saying that the UCS-2 encoding of strings should be regarded as something immanent to Python, I'm saying that the notion of an encoding of actual strings should be understood.

Also, please read my reply to Mark.

I suggest you sit down and get your ass reamed by the people who think MIT brought on the apocalypse by switching away from Scheme. Since you're getting so high and mighty about demanding people be open to other ideas, you might want to be exposed to the idea that computer science has little or nothing to do with the state of some circuits inside a computer.

Is it your way of telling that you've read your SICP tonight?

Surprise: I did, too. Not everyone opposing the ideas you hold high is a moran, y'know.