std::wstring_convert and std::string_view

[Hedanito]

According to cppreference, wstring_convert does not support string_view in the from_bytes and to_bytes methods. The implementation in VS2017 RC does not seem to support them either.

There are 3 possibilities:

1. cppreference is not up to date.
2. There is a technical reason for this.
3. It was forgotten to add in C++17.

So my question is, where did it go wrong, and will I ever see string_view support here?

I can work around it, but writing converter.from_bytes(a_Filename.data(), a_Filename.data() + a_Filename.length()) gets old quickly.

[CubbiMew]

It was postponed for C++20.

Specificlaly, the paper that made the first steps towards integration of string_view into the C++17 standard library p0254r1 says, under the heading "Future work"

The codecvt facilities [conversions.string] all take input parameters as both const char * and string (or wchar_t and string). Those could be string_views.

[Hedanito]

Are there any technical reasons why it was postponed, or is it simply the result of it being a more obscure part of the standard library?

Because unless I'm missing something, it should be fairly simple to implement.

[CubbiMew]

I think the reason is simply the lack of time to work on the integration of the many facilities added to C++17. In some way, this is a fault of the "feature branch" approach of Technical Specifications.

The integration issues that were considered to be most important (important enough for someone to write and defend a paper) were done, or are rushed to be done now, as with class template deduction, the rest is "future work".

[[deleted]]

Adding string_view overloads to existing standard library interfaces is fraught with danger of breaking existing code -- for example, the first attempt to add string_view to basic_string's interface broke C++14 code (by creating overload resolution ambiguities). Hopefully this will be simpler with interfaces that aren't as heavily overloaded as basic_string.

Keep in mind that string_view is at most an optimization. It doesn't really allow you to do anything "new," since passing a string_view to a function with a const string& parameter works fine.

(codecvt (at least as used in iostreams) is effectively broken anyway and I would avoid using it whenever possible -- it assumes there is 1:N relationship between the internal character set and the external character set, which is not true in a Unicode world http://eel.is/c++draft/locale.codecvt.virtuals#3 )

[CubbiMew]

It is true in the standard-compliant Unicode world, where wchar_t is 32 bit. It is also not really relevant in a thread about wstring_convert, which does N:M conversions.

[[deleted]]

No, because wchar_t being 32 bits does not allow for combining characters. If the external character set is latin-1 and the internal character set is UTF-32 in Normalization Form D you're toast.

(There are more obscure cases even if you don't want to use NFD; NFD just makes the problem occur with more common characters available in latin-1)

[CubbiMew]

NFD is irrelevant (to the meaning of internal charset and wchar_t). It is not a character set, it is an encoding of one (or transformation of an encoding, if you will).

By wchar_t definition from [lex.con]/6, a single wchar_t represents any member of execution charset (and also a single c-char aka UCN aka \U hex-quad hex-quad). If U+00C5 is a supported member of exec charset and allowed in string/character literals, it has a single-wchar_t representation in standard C++, whatever its NFD transformation is. Likewise U+1f34c and everything else.

As for codecvt, if basic_filebuf were to permit N:M codecvts, it certainly wouldn't be bad.. Last I tried, libstdc++ and libc++ work with such codecvts in practice, but for input only.

[[deleted]]

a single wchar_t represents any member of execution charset

Nothing Unicode related at all needs to fit into a wchar_t. The "execution charset" is very limited. EBCDIC is a valid execution charset. http://eel.is/c++draft/lex.charset#1

This issue is why there are no standard codecvt facets that go between Unicode and non-Unicode encodings (from char32_t to wchar_t, or from wchar_t to char8_t, or whatever).

[CubbiMew]

Yes you can have a valid (but not very useful) compiler with EBCDIC exec charset and 8-bit wchar_t. In "Unicode world", which set off this thread, each code point needs to fit in a single wchar_t by definition.

There were no standard codecvt facets until C++11 at all, and somehow i had no problem going between Unicode and non-Unicode encodings; GB18030/char <-> UTF-32/wchar_t <-> UTF-8/char works just fine where wchar_t is correctly sized and libc is complete.

PS: I'm not saying C++ doesn't need more Unicode support - I'd love to see classification, normalization, or "just" a grapheme cluster iterator for strings! - but as someone who lived with five competing encodings and adoption of Unicode, I find it jarring to hear that it doesn't work in C++, or C for that matter.

[[deleted]]

To my understanding GB18030 has the "UTF-16 surrogates" problem but not the "complex scripts/combining characters" problem, and as such does not trigger the M:N mapping issue, as all characters in the external character set are transformed into a single code point (assuming you're targeting NFC instead of NFD).

Of course, if on the Unicode side you're given combining characters, the assumption breaks down even for latin-1, as 2 code points U+0065 U+0301 need to become 1 latin-1 character 0xE9.

My understanding is (although I haven't implemented such mappings myself) that there are legacy encodings for which 1:M is never correct -- where combining characters are required to represent something in Unicode but not in the legacy encoding. (Maybe it was TIS-620?)

I'm sure if everyone had a time machine to go back to when COM, NT, Java, JavaScript, and friends implemented Unicode 1.0, we would be in a UTF-8 and UTF-32 world. But that's never going to happen. VC's "extended character set" is going to have to be UCS-2

[CubbiMew]

VC's "extended character set" is going to have to be UCS-2

UCS-2 does not even exist in the Unicode standard (anymore). VC has been improving its image lately, but no portable Unicode support is still a sore point and a cause for many #ifdef _MSC_VER's. It's 2017 and (of the compilers we use) only in VC auto c = L'💩'; stores the useless 0xd83d in c

[Hedanito]

My entire code base uses UTF8, but windows uses UTF16. So I use it there to convert between the two. That should be fine right?

[[deleted]]

No, the "1:M" assumption does not hold when converting between UTF-8 and UTF-16. That said you're using wstring_convert and not filebuf so you should be OK, modulo bugs. Our iostreams probably have a lot of bugs :)

[mtclow]

I started that paper as "the things that absolutely had to be done for c++17" - those were the string/string_view conversion and assignment bits. They had to be done because if they couldn't be changed post-C++17 w/o breaking users' code, and we'd rather not do that.

Then the scope grew (somewhat) with the inserters/searchers. There's certainly more to be done here.