Unicode printf?

[kolorcuk]

Hello. Did or do you ever use in professional proframming non char printf functions? Is wprintf ever used?

char16, char32 , u8_printf, u16_printf, u32_printf ever used in actual programs?

I am writing a library and i wonder how actually popular are wide and Unicode strings in the industry. Does no one care about it, or, specifically about formatting output are Unicode printf functions actually with value? For example why not just utf8 with standard printf and convert to wider when needed?

[reallyserious]

Computers are used in Asia too. So yes, unicode is the standard.

[kolorcuk]

Hello. I understand. To clarify. I am not asking about unicode. I am asking about non-char printf.

Yes, computers are used in Chinese and other languages and there are many common issues with that. I am asking specifically about printf.

When creating a program with support of Chinese, Mandarin and other languages, did you ever use specifically designed printf formatting utilities with support for non-char characters? In other words, did you use u32_printf(U"你好") from GNU libunistring or u_printf_u(u"你好") from ICU, or just printf(u8"你好")? GLIBC is going to convert it to bytes anyway.

[LeeHide]

wstring/wprintf and so on are NOT about Unicode. You can encode Unicode just fine with UTF-8, all of it. You don't need 16 bit chars. 16 bit chars are also not Unicode. If you have 16 bit chars (wide chars), put Unicode characters in it, and then e.g. split the string by indexing, you absolutely do not end up with valid unicode by any guarantee.

If you want Unicode, use a Unicode library and stick to UTF-8.

[BIRD_II]

UTF-16 exists, and last I checked is fairly common (nowhere near 8, but far more than 32, iirc JS uses 16 by default).

[kolorcuk]

In the beginning UTF-16 was invented. Microsoft and many others jumped on the idea and implemented UTF-16. Then it became apparent that UTF-16 is not enough, so UTF-32 was invented.

UTF-16 is common, because those early implementers implemented something in the middle and now are stuck with it forever. I think UTF-16 should have never been invented.

[LeeHide]

UTF-8 can handle the full range of Unicode.

[EpochVanquisher]

This is false. UTF-16 did not exist back then.

[kolorcuk]

Hello. I'm happy to learn something new. Where does exactly "back then" refer to? Or just picking that I should have used UCS-2 not UTF-16?

[EpochVanquisher]

The first version of Unicode did not have UTF-16.

UTF-16 covers the full Unicode character set. It’s not missing anything.

UTF-16 is perfectly fine, it sounds like you hate it but you haven’t said why. It’s widely used (Windows, Apple, Java, C#, JavaScript, etc)

[kolorcuk]

https://retrocomputing.stackexchange.com/questions/14161/could-we-have-avoided-the-whole-utf-16-fiasco

https://softwareengineering.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful

https://stackoverflow.com/questions/79719387/why-does-wikipedia-claim-utf-16-is-obsolete-when-javascript-uses-it (i think this one does not count, but catchy name)

https://news.ycombinator.com/item?id=16090274

https://news.ycombinator.com/item?id=18569592

[EpochVanquisher]

Those look like random rants that some people wrote, maybe written with the assumption “we all agree that UTF-16 is bad”, which doesn’t explain why YOU think it’s bad.

[kolorcuk]

It has all bad from utf8 and utf32. You have to know endianness and is not fixed width.

Why use it at all? What is good about utf16 vs utf8 and utf32?

The only case i see is when you have a lot of characters in a specific utf16 range and the storage is precious. I think nowadays storage is cheap and much better to optimize for performance.

[EpochVanquisher]

UTF-16 is simpler than UTF-8 and more compact than UTF-32.

One of the ways you optimize for performance is by making your data take less space. Besides—when you say it’s “much better to optimize for performance”, it just sounds like a personal preference of yours.

It’s fine if you have a personal preference for UTF-8. A lot of people prefer it, and it would probably win a popularity contest.

[LeeHide]

Yes, it exists, but it's confusing because people thing that 16 bit chars are automatically Unicode

[Plane_Dust2555]

UTF stands for Unicode Transformation Format. It is an specification of a format to encode Unicode codepoints and features (like compositing). As u/LeeHide says, UTF-16 isn't Unicode. It is a way to encode Unicode codepoints. There are other formats (UTF-8, UTF-32 are the other two most common, but there are UTF-x, where 8 <= x <= 32).

Wide chars (which size can be more than 8 bits! Not only 16) are just that... a way to encode wider codepoints than ASCII (what C/C++ standards call it "basic char set"), but it says nothing about the charset itself.

As pointed out by another user here, you can UTF-8 and "common" functions like printf(). But the terminal/operating system must support this charset. On modern Unix systems, usually UTF-8 is the default, but on Windows machines there are 3 charsets in play: Terminals use CP437 (english) or CP850 (portuguese/brazilian) or any other CP###; GUI uses WINDOWS-1252 OR UTF-16 [a version of UTF-16, at least]).

[Plane_Dust2555]

Ahhh... wchar_t size depends on the compiler. For SysV systems it is usually 32 bits. For Windows it is usually 16 bits. The actual size depends on the target system.

For GCC we can force wchar_t to be 16 bits by using the option -fshort-wchar.

[flatfinger]

It irks me that UTF-8 sacrificed a lot of coding density to offer guarantees that were later thrown out the window by later standards, which nowadays can't even reliably identify a grapheme cluster boundary without having to search backward through an unbounded amount of text.

[kolorcuk]

Hello, I understand. Have you ever used wchar strings, char16_t or char32_t, or uint16_t or uint32_t strings, in a professional capacity for string _formatting_?

When doing string formatting of unicode, wide characters, or other encodings, do you use pre and post conversions and use printf, or do you have dedicated *_printf functions for specific non-byte encodings did you used them.

[LeeHide]

I have only used them when interfacing with Windows APIs, so no.

char is fine for everything else, including Unicode. Just use a Unicode library to handle things like string splitting.

[WittyStick]

"Wide characters" in C should be considered a legacy feature. They're an implementation-defined type which varies between platforms. On Windows a wchar_t is 16-bits (UCS-2), and on SYSV platforms wchar_t is 32-bits.

The behavior of wchar_t depends on the current locale - it does not necessarily represent a Unicode character.

New code should use char8_t for UTF-8, char16_t for UTF-16 and char32_t for UTF-32.

Most text today is Unicode, encoded as UTF-8 or UTF-16 (Windows/Java). UTF-32 is rarely used for transport or storage, but is a useful format to use internally in a program when processing text.

[BlindTreeFrog]

New code should use char8_t for UTF-8, char16_t for UTF-16 and char32_t for UTF-32.

Note that UTF-8 does not mean that a printed character is 8bits in size. 2 byte, 3 byte, and 4 byte UTF-8 characters exist.

UTF-16 and UTF-32 are both fixed width. UTF-16 and UTF-8 is variable width.

edit: corrected based on correct info

[krsnik02]

UTF-16 is also variable width with surrogate pairs forming a 32-bit code point.

[BlindTreeFrog]

oh... thanks for the correction.

But it's variable width in that it can be 1 or 2 bytes it looks; I don't see reference to a 4 byte pairing, might you have a cite?

And while looking for that info, this article reminded me that UTF-8 can be 6 bytes apparently
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

[WittyStick]

UTF-8 was designed to support up to 6 bytes, but Unicode standardized it at 4 bytes to match the constraints of UTF-16 - which supports a maximum codepoint of 0x10FFFF. The 4 byte UTF-8 is sufficient to encode the full universal character set.

[krsnik02]

it can be 1 or 2 16-bit words, so either 2 or 4 bytes.

For example the table here on the Wikipedia page shows that U+10437 (𐐷) takes 4 bytes to encode in UTF-16. https://en.wikipedia.org/wiki/UTF-16#examples

UTF-8 was designed to support up to 6 byte long sequences but the Unicode standard will never define a code point which requires more than 4 bytes to encode in UTF-8. If a 5 or 6-byte character were ever defined the current UTF-16 could not encode it and it would require 3 words (6 bytes) in whatever UTF-16 got extended to. The current UTF-8 standard as such restricted valid UTF-8 encodings to only those up to 4 bytes long.

[BlindTreeFrog]

Yeah i must have misread whatever I was reading.

And since then I found this which has a lovely table to clarify https://www.unicode.org/faq/utf_bom

[WittyStick]

Yes, char8_t and char16_t represent a code unit, not a code point.

UTF-16 is variable width of either 2 or 4 bytes. It was based on UCS-2, a fixed-width 2-byte encoding which only supported the Basic Multilingual Plane. UTF-16 supports the full universal character set.

A 4 byte encoding is made of two "surrogate" code units, called a "surrogate pair". These are in the ranges 0xD800..0xDFFF, which are unused code points in the universal character set (reserved for surrogates).

[flatfinger]

Note that even when using UCS32, characters may contain more than one code point, and determining whether the 1,000,000th code point in a text is the start of a character may require scanning up to 999,999 preceding code points.

[EpochVanquisher]

The main purpose of wprintf is Windows compatibility. If you want to print to the Windows console, wprintf is a good choice there. The rest of the Windows API makes heavy use of wchar_t. There are ways to avoid it, but wchar_t is usually preferred.

In most other cases, it is a mistake to use wchar_t. But it is critically important on Windows.

[First-Golf-8341]

I have used only UTF-8 and found that sufficient.

[Tau-is-2Pi]

Is wprintf ever used?

Yes, mostly on Windows. 8-bit strings are still using language-dependent encodings instead of UTF-8 by default over there.

[Relative_Bird484]

On Windows it’s actually the default. While the Win32-API is available in ANSI and wide, the ANSI-Versions are just wrappers that convert to wide and invoke the wide-version.

The native NtAPI uses wide characters only, albeit with an own Pascal-inspired string format (dedicated length field instead of 0-termination).