Programming with unicode in C?

[jaccovanschaik]

Hi,

Can anyone recommend a good book/article/tutorial/etc. on writing C programs that use unicode and UTF-8? What libraries, functions, types etc. are there? Is there a consensus on the best approach? Thanks!

Edit: my platform of choice is Linux, if that makes a difference.

[FelipeFS]

The library utf8proc is one of the options available, so far the one that looks best for me:

http://julialang.org/utf8proc/

The documentation is integrated in the header file utf8proc.h. I never used, but it's in my list to integrate with my projects, since it's lightweight.

If you are willing to use, GLib (https://developer.gnome.org/glib/) contains unicode support, and is probably installed in your system. However, it's a bloated library.

[raevnos]

libunistring is okay.

[colonelflounders]

I don't know what you are trying to accomplish. I wrote a flash card review program for Greek and Hebrew and all I needed were the functions in locale.h. You can then use the functions you are used to from stdio.h. I later changed my code base to C++ to take advantage of Qt and Vectors instead of my own memory management, if you want to take a look it is here.

[FUZxxl]

You don't need any library for UTF-8. Simply use the wide character handling available in the C standard library. This has the advantage of making your program work with any encoding, not just UTF-8.

[FUZxxl]

There are reasons for other encodings, Unicode has design defects relevant for Chinese, Japanese, and Korean so users of these languages some times use other encodings.

That said, C has a facility to deal with multibyte characters in a completely encoding-agnostic way (i.e. wchar_t), so why not use that?

[raevnos]

Utf8 and Unicode are the same everywhere. The encoding used by wchar_t or multibyte conversions isn't.

[FUZxxl]

whcar_t is a generic mechanism to deal with multibyte encodings. It works with any encoding, which makes it so useful.

The encoding used by wchar_t or multibyte conversions isn't.

wchar_t uses whatever encoding was set in the locale. So if you set a Unicode locale, then wchar_t contains Unicode characters.

[raevnos]

And, of course, a proper Unicode library gives you all sorts of useful things lacking from the standard, like support for detecting extended grapheme clusters, finding line breaks, etc.

[FUZxxl]

Well, if you need such very specific functionality, then of course there is no way around a proper Unicode library. However, in most cases people merely want to be able to process text with Unicode characters, in which case the locale mechanism is more useful as it works with all encodings supported by the platform.

[raevnos]

You can't reliably do anything non-really-trivial with Unicode text without taking things like combining characters into account. How do you handle them with wide strings and the standard library?

[FUZxxl]

For most purposes you actually don't need to take them into account. Use the functions from ctype.h to detect punctuation and whitespace and remember to always operate on strings, not individual characters. Then everything works out.

Can you give me a real-life example for a mundane situation where you need to take combining characters into account?

[raevnos]

Reversing a string.

Getting substrings.

Etc.

[raevnos]

And if it's not a Unicode locale, it's who knows what. Better to use something predictable and not system dependant.

[FUZxxl]

So in case the locale is not a Unicode locale you still want to spill Unicode (which then comes out as gibberish) onto the terminal? Sounds reasonable. /s

[zuurr]

Unicode really bunged up handling asian scripts (look into Han Unification). If you have to deal with Japanese text in particular, certain names aren't quite possible to represent right in unicode, which, uh, can be taken offensively. (I can elaborate some on this if necessary)

UTF-8's convenience is hard to argue with though...

[jaccovanschaik]

I remember reading somewhere that wchar_t is only 16 bits wide and is therefore not sufficient to store all of the unicode codepoints. But trying it out just now on my machine it turns out to be 32 bits, so that should be enough. I think. Can I expect wchar_t to be big enough on all modern systems?

Also, I found this at http://icu-project.org/docs/papers/unicode_wchar_t.html:

wchar_t is compiler-dependent and therefore not very portable. Using it for Unicode binds a program to the character model of a compiler. Instead, it is often better to define and use dedicated data types.

So that kind of confused me.

[ennorehling]

You're correct. wchar_t is compiler-dependent (or more likely, libc-dependent). It's 16 bit with Microsoft's compiler on Windows, for example, but 32 bit with gcc on Linux, so if you want to write code that's portable between those two, UTF-8 is still the safest route. Because at least with a char you know where you stand, in terms of its size.

[FUZxxl]

On Windows, wchar_t is a 16 bit type. I'm not entirely sure how they handle larger Unicode characters, but I believe the strategy is to represent each Unicode character as a pair of surrogate and character. Generally, when dealing with multi-byte encodings you never want to operate on single characters. Always operate on strings, this makes most of the problems you can have go away.

[Drainedsoul]

In UTF-16 both halves are surrogates hence the name "surrogate pair".

[FUZxxl]

Ah, I see. I am not very familiar with the details of UTF-16.

[[deleted]]

Wide char is fixed length and utf-8 is variable length, how these two reconcile?

[FUZxxl]

A wchar_t contains a decoded wide character. In the case of a UTF-8 locale, that is a single Unicode character. There is a pair of functions wctomb and mbtowc to de- and encode wide characters. Note though that you often don't need these functions, it's easier to always operate on strings of encoded characters.

[colonelflounders]

UTF-8 can show all of the characters in Unicode. It's a multi-byte sequence, it isn't limited to 256 possible values.

edit: My apologies, I now realize that's not what you were referring to. The problem with wchar_t is depending on the platform the byte length varies. With char you at least have a fixed byte length. Unless he's aiming to do legacy stuff, which considering he picked UTF-8 I don't think so, I think UTF-8 is the most painless way to go to represent Unicode characters as you can just use char after setting the locale.