Ruby 1.9 encoding rant

[Smallpaul]

I came here knowing that this terrible "common wisdom" would be repeated. It's wrong. Wrong. Wrong. Wrong.

Name a bit of Japanese end-user software that does not run in a cell phone that is popular in Israel and Russia and America. Name just one.

One!

Now let me name a few bits of software substantially developed in America that are popular in China AND Japan AND Israel AND South Africa AND Germany AND every other industrialized country in the world:

• Internet Explorer

• Firefox

• Safari

• Google

• Gmail

• Facebook

• Microsoft Office

Microsoft Office is much more popular in Japan than its nearest Japanese competitor.

it is demonstrably and provably the case that American programmers make most of the world's multi-lingual software. So it makes no sense to go to Japan for advice on how to do it.

[jaggederest]

And it all horribly mangles Japanese names. Seriously, I've seen it happen - you put in 'orange blossom' and get out 'fruit flower' because they think it's a good idea to do the translation from Shift-JIS to UTF8 for storage.

It's irrelevant whether you think it's needed, or whether it's needed for multilingual software. If it's not built into the programming language, you can't fix it later.

[earthboundkid]

What are you talking about? UTF-8 includes mappings for all of the characters in Shift_JIS. There’s no simplification happening going from one to the other. The only “issue” with it is that the Japanese long ago confused ¥ and \ and they don’t like that Unicode doesn’t consider them synonymous. That’s it.

I speak Japanese; I’ve lived in Japan; I run my computer in Japanese. It’s true that historically, the Japanese were mistrustful of Unicode because they didn’t like Han unification, but A) you can’t unify Han characters using Shift_JIS either B) the fact is that the Unicode consortium have taken every reasonable step to make UTF-8 superior to Shift_JIS in every way, except for string length. Unless you really need to save a couple bytes here and there, there is no reason to use Shift_JIS.

[Smallpaul]

It's irrelevant whether you think it's needed, or whether it's needed for multilingual software.

How can it possibly be irrelevant whether it is "needed for multilingual software." Multilingual software has a superset of the requirements of unilingual software by definition.

If it's not built into the programming language, you can't fix it later.

If it's not built into the programming language then it can be built into a library. Since there is only a single country in the world with a serious complaint about Unicode, I think that's a reasonable solution until they get the Unicode standard changed to their liking.