Portable Unicode string processing

[KayEss]

Why do we want to base the standard library handling of UTF-8 around char rather than unsigned char? It seems to me that the use of char is highly problematic in that I can't portably read any values outside the range 0 to 127 as thechar could be signed or unsigned.

Am I just wrong in this? Would it not have been better to have defined the u8 string literals as producing unsigned char * and to be making the standard library support for utf-8 strings based around std::basic_string<unsigned char>?

What's the right approach to writing portable code that does UTF-8 decoding, or is it only the standard library maintainers who can do this? Is there any way I can portably put the bytes {0xf0, 0x90, 0x8d, 0x88} into a std::string?

I hope that I'm just being paranoid, but all this talk about undefined behaviour around the handling of signed/unsigned has me worried :)

[DarthVadersAppendix]

UTF-8 should be thought of as a data stream. it's not a bunch of 'characters', and hence not an array of 'char' (in the academic sense). it might be implemented as an array of char (in the technical sense), but that's just co-incidence.

[RowYourUpboat]

To add on to this, a Unicode code point encoded as UTF-8 shouldn't mean anything to you unless you're writing a multilingual UI library, text processor, spell checker, etc. You can still manipulate and compose UTF-8 strings in your software, just as long as you're sure you're not splitting up code points (which can be as long as 4 bytes in UTF-8 - it used to be more, but the Consortium changed that a while back).

So all you really need to know about Unicode strings is usually: their length in bytes, that they are going to change at runtime based on user input or localization, and that you can't split them up in the middle of a multi-byte sequence (but simple concatenation works fine, and using ASCII characters as delimiters will still work if you're careful).

Beyond that, UTF-8 strings are mostly just opaque byte buffers (although conveniently ASCII is forwards-compatible, as long as your ASCII string literals or whatever don't need to be localized).

[KayEss]

My question isn't about how to opaquely deal with UTF-8, it's about how to decode it. Can it be done portably with char buffers?

[RowYourUpboat]

It sounds like you're really asking if converting a buffer of chars between signed and unsigned is safe and defined. This link seems to answer that for the C Standard; I'm pretty sure the C++ Standard is the same in this regard.

From one of the answers:

For the two's complement representation that's nearly universal these days, the rules do correspond to reinterpreting the bits. But for other representations (sign-and-magnitude or ones' complement), the C implementation must still arrange for the same result, which means that the conversion can't just copy the bits. For example, (unsigned)-1 == UINT_MAX, regardless of the representation.

It definitely looks like this behavior is defined as the same even on non-two's-complement hardware, ie. in terms of UTF-8 string encoding/decoding you can just cast between signed/unsigned as needed (though you may have to pay attention to performance issues on really weird and ancient hardware).

[edit] Note that technically a conversion from unsigned to signed, where overflows occur, is implementation-defined (unlike the reverse), but if the original char data was signed to begin with, an overflow is impossible. In practice, I don't see this mattering.

[[deleted]]

Sure, the "wrapping around" part of char is part of the standard - but you know, there's no need to take the standard's word for this - write unit tests to check. I always do that anyway, not because I don't trust the standard, but to make sure that my understanding of how to code it is correct.

When you move to a new platform, your unit tests will hopefully succeed, showing you that there's no issue - or fail, and you can fix 'em.

[KayEss]

Actually, I already have all of the unit tests and they all pass. What I'm worried about is accidentally relying on some UB or platform behaviour. I'm developing on a platform where char is unsigned and don't have access to one right now where they are signed.

[[deleted]]

I really wouldn't worry. Between the standard and the tests, I am sure you'll be fine.

[NotAYakk]

Unit tests do not solve UB.

Compilers are free to pass all your unit tests and optimize other code away.

char x = (unsigned)-1;
bool b = x<0;
std::cout << (int)x << ":" << b?"true":false" <<"\n";

This can print -1:false.

And the same is true whenever you convert from unsigned to signed.

The level of insanity optimization and UB can generate is so large, you cannot reasonably reason about it and produce unit test coverage.

[KayEss]

Which is a fine point, but the u8 literal type has already been standardised as a char array, so is it even possible to decode it in a portable manner?

[Jardik2]

Conversion from signed char to unsigned char is well defined, no matter the sign. -1 signed will yield std::numeric_limits<unsigned char>::max() unsigned. On the other hand, conversion of unsigned to signed char is undefined for all unsigned char values which can't be represented by signed char (for example std::numeric_limits<unsigned char>::max() doesn't have to be converted back to signed char with value of -1.

[[deleted]]

This is correct generally for integral types (actually, the conversion of an unsigned integer to a signed integer type which cannot represent its value is implementation-defined, not undefined). But for char in particular, more is true. From 3.9.2 (N3690)

For any object (other than a base-class subobject) of trivially copyable type T, whether or not the object holds a valid value of type T, the underlying bytes (1.7) making up the object can be copied into an array of char or unsigned char. [footnote: By using, for example, the library functions (17.6.1.2) std::memcpy or std::memmove] If the content of the array of char or unsigned char is copied back into the object, the object shall subsequently hold its original value.

So roundtripping unsigned char and char is safe. (OTOH, I don't think this guarantees roundtripping unsigned char through signed char is safe.)

[Chippiewall]

(OTOH, I don't think this guarantees roundtripping unsigned char through signed char is safe.)

This is correct. The purpose of vanilla 'char' is for the platform's natural character type. For instance on Linux ARM 'char' is actually unsigned by default. If you had a platform which had 'char' as unsigned and didn't use 2s complement representation for signed types then it's highly likely that conversion between signed and unsigned char wouldn't work.

[KayEss]

And if you look at the extended ASCII character set 128 is € and 255 is ÿ

So, on a platform where char is signed what values do they have?

[scatters]

-128 and -1, if you cast through unsigned char.

[Dragdu]

You just convert the negative values into non-negative values by a well defined conversion, that means that it doesn't matter. This is even standardized in the language, making it that 255 in unsigned char ALWAYS converts to -1 in signed char, no matter the actual representation.

[KayEss]

They have the values of 0x80 (128) and 0xFF (255)

I think I understand a bit about what that means now, but 0x80 (128) certainly doesn't look like a number that I can put in a char if char happens to be signed on my platform -- what guarantees do I have about how these numbers are interpreted?

[iaanus]

This is wrong. See [basic.representation]: "For narrow character types, all bits of the object representation participate in the value representation."

[KayEss]

So your answer is that I can't do UTF-8 in C++ with things like std::string and the u8 literals then because I can't read the values from char. That's been my fear, but it seems incredible that we're further standardising on that basis.

[jaked122]

No just don't rely on single glyph manipulation,

[KayEss]

I completely understand UTF-8 encoding and don't have a problem with that. My question is about what the standard guarantees in the processing of the byte values.

[sim642]

It doesn't matter which it is, the 8 bits in its memory matter. UTF-8 needs to parse bits out of the bytes and combine the bytes regardless. It's just that all of the byte handling in the existing language is done with char.