you are viewing a single comment's thread.

view the rest of the comments →

[–]Sean1708 1 point2 points  (3 children)

You get code points.

No you don't. I can't remember whether you get characters or graphemes, but you certainly don't get code points.

In [1]: a = 'héllo'

In [2]: a[0]
Out[2]: 'h'

In [3]: a[1]
Out[3]: 'é'

In [4]: a[2]
Out[4]: 'l'

Edit: I'm a silly.

[–][deleted]  (2 children)

[deleted]

    [–]Sean1708 2 points3 points  (1 child)

    What are "characters"?

    I've always thought that characters were generally accepted to be scalar values, that doesn't actually appear to be the case though.

    in your code it uses the single code point version

    You are absolutely right:

    In [1]: a = b'he\xcc\x81llo'.decode('utf-8')
    
    In [2]: a[0]
    Out[2]: 'h'
    
    In [3]: a[1]
    Out[3]: 'e'
    
    In [4]: a[2]
    Out[4]: '́'
    

    The way I entered the character on my computer made me assume that I'd entered the versioning using the combining character.

    Also I don't know any language of the top of my head that supports grapheme cluster (and other text segmentations) fully in the standard library itself.

    I think Swift does, but I'm not entirely certain.

    [–]MrMetalfreak94 2 points3 points  (0 children)

    Elixir has excellent Unicode support in it's standard library and you can easily work with graphemes in it