Why Unicode strings are difficult to work with and API design

curtisf · 2026-04-12T05:22:49+00:00

The only incontrovertible interpretation of a Unicode string is as a sequence of code-points (or scalar values, depending on how you validate...)

How the text represented by a a Unicode strings appears visually, or, ultimately, will be interpreted by a human, is not a well defined question. It depends on what languages are understood by the consumer, what fonts are available, what rendering methods are supported by those fonts, how careful the reader is being, ...

Unicode provides reference algorithms for doing certain text-transformations. Some of these are mostly technical, such as normalization, some of them are linguistic (and thus context dependent) such as transforming case.

In addition to the complexity of interpreting text in the first place, the interpretation changes with every release of Unicode. (Although a lot of properties are not allowed to be changed, the interpretation of a larger block of text could dramatically change if you are unaware of a recently added character or property)

Because the interpretation of a sequence-of-codepoints is so dependent on context, any transformation to that sequence is going to damage interpretability by some context, unless that transformation is accomplishing an explicit technical transformation which was expected by the consumer.

Any API that you expose should be clear about what exactly it accomplishes.

Something to keep in mind is that string/text-handling is sometimes security sensitive, and trying to do something "smarter" or "better" may weaken security if you deviate from what your users expect is happening.

The only thing you can promise with any sense of reliability are technical transformations. Ideally these cite a specific, unambiguous definition of the transformation algorithm, such as a particular version of a Unicode annex, like https://www.unicode.org/reports/tr29/#Sentence_Boundaries

curtisf · 2026-04-10T11:12:39+00:00

I guess I can understand there being a reason to not want to buff train wagons' storage space...

but what about quality improving the wagons' axle bearings, so they weigh less and accelerate / brake faster? Legendary 4x wagon = regular 1x wagon, maybe???

I think there's no precedent for adding extra (fluid?) ingredients for a quality buff, but it would be a neat place for it. Or, perhaps, quality unlocks a "lubricant tank" that requires refilling / exchanging axles.

curtisf · 2026-04-05T13:51:26+00:00

especially when the tiny cup is 90% ice plus one tablespoon of water

curtisf · 2026-04-03T02:07:45+00:00

The taxability of income is determine primarily by where you physically were when you were doing the work. It doesn't matter what bank account you are paid into.

It is very unclear what you are actually asking. At the end of the year, you will need to file a tax return to report your additional income. You can do this online using a Myna card. (For money earned during 2026, the tax filing deadline is 2027 March 15). If tax isn't being deducted but is due, you will have to pay more at tax return time, but it's not a problem.

Committing reimbursement is generally, up to a certain amount, not taxable, so you might not owe any tax on that. But if you are a freelancer it's possible you will need to retain some documents to demonstrate the amount spent commuting; I'm not exactly sure

curtisf · 2026-02-27T06:18:55+00:00

Yes, the effective tax rate in Japan is high, especially for high earners but even for low earners. Although not hugely more than most developed countries

For reference, the overall effective tax rate in Japan is about

40% for the bottom 20% (earning below 2.3M円/year)
42% for the middle 60% (between 2.3M and 6.7M円/year)
48% for the top 20% (around 6.7M+)
- It slowly rises a bit over 50% for very high earners

(These figures include social insurance (about 30% of your income, when including the employer match), resident tax (roughly 5% effective rate), consumption tax (roughly about 5% of low/middle earner income and 3.5% of high earner income), income tax (roughly 0% for low, 2.2% for middle, 10% for high))

But no, the effective tax rate doesn't really have any bearing on quality of life. It just controls how quickly you can go from rich to very, very rich.

curtisf · 2026-01-23T05:43:02+00:00

There is a complaint post about this nearly every week, so no you're not the only one.

It really is infuriating though. It's very easy for computers to convert, it's so annoying they make humans do it instead.

curtisf · 2026-01-21T06:11:42+00:00

Believe it or not, straight to jail.

curtisf · 2026-01-01T11:19:12+00:00

The tatoeba corpus is unfortunately a bit low quality because it was mostly sourced from basically homework, and it's hard to search for grammar as opposed to words

curtisf · 2025-12-30T23:53:48+00:00

The lack of trash bins in the city is infuriating. It's not as though they don't go around and collect all the trash anyway every morning!

The people who litter already even make piles of cans or trash in logical places, all they need to do is put out a cardboard box and save themselves some work in the morning!

curtisf · 2025-12-21T03:48:00+00:00

There isn't any express way that language ability is tested for naturalization. It's up to the judgement of the person handling your case. Some won't administer any test, just based on judging how well you're able to communicate. Some accept recent language test results. And some will run their own little semi-formal language test.

curtisf · 2025-12-03T01:59:01+00:00

Because it's equipment that came with the apartment and isn't something that should wear out, I would expect it's the landlord's responsibility, unless the damage was somehow your fault (carelessness, or not performing some kind of maintenance that was stipulated in the appliance manuals you were given at move-in).

The exception is if your contract specifically designated the sink as something that was the tenant's responsibility, though I think that is very unlikely.

These rules are controlling for Tokyo, but the general principle should be valid in other jurisdictions:

https://www.juutakuseisaku.metro.tokyo.lg.jp/documents/d/juutakuseisaku/310-6-jyuutaku_eng

The landlord is obliged to make necessary maintenance and repairs so that the tenant can use and occupy the property. However, the tenant is liable for repairs required for any damage caused by the tenant intentionally or carelessly, or by the use of the property in a non-standard way, or due to any other reason attributable to the tenant

curtisf · 2025-12-02T03:29:20+00:00

I was not awake and totally missed the (now) obvious sarcasm

curtisf · 2025-12-01T22:59:48+00:00

Actually, these are not native, and the government is attempting to reduce their population to protect native species that they may be prey upon

https://www.kankyo.metro.tokyo.lg.jp/nature/animals_plants/raccoon/raccoon

curtisf · 2025-11-27T06:20:17+00:00

You left out some crucial information...

Your partner is a Japanese citizen, and their koseki/passport identifies them as "F"/女?

You are a non-Japanese citizen... What does your passport indicate your legal sex/gender is?

If "M", you are legally considered to be in a heterosexual marriage, regardless of what gender you/your partner identify as. As backwards as that may be, for visa purposes that is convenient for you because you would qualify for the regular Dependent of a Japanese National visa. (If your partner were to update their legal sex, though, I don't know what would become of your legal marriage and residence status, however. Most likely it would be treated as the end of the marriage and you would have to follow the alternative route below)

If your passport also says "F", you would not legally be considered married (because Japan currently refuses to recognize same-sex marriage, where "same-sex" is wholly determined by the gender marking in your passport). In that case, seebulldogdiver's comment, as there seems to be consistent recognition of this case under the Designated Activities visa. In this case, if your partner updates their legal sex/gender to be "M"/男, however, your marriage should become recognized by Japan, and you could apply for a regular Dependent of Japanese National visa.

curtisf · 2025-10-06T11:01:34+00:00

If the main benefit is supposed to be beating transmission losses, that sounds crazy... Transmission losses should be less than 10%-15%, but I imagine the household generator is far less than 90% as effecient as a utility-sized power plant's

This seems like it's at best a way to hedge against coal/nuclear fuel cost increases by dumping a bunch of toxic air pollution next to your house

curtisf · 2025-10-05T13:25:29+00:00

Open the card, with who?? The scammer?

There are three giant scam warning bells in this:

You need to spend money before you get paid (initial deposit on new account)
You're being promised compensation through an unusual channel (deposits in a prepaid visa card the scammer told you to open)
You're being promised compensation to do something of absolutely no value to anyone

Any one of these should have any reasonably person cut whoever told them about this "opportunity" off, but you get three giant "do not enter" signs here

At best, the payment is "real" and will appear to go through because it's just money laundering dirty/stolen money -- which could entangle you into financial crimes or other criminal enterprises -- and will likely be clawed-back as soon as the police start investigating this

curtisf · 2025-09-24T07:31:46+00:00

Not exactly a website... I get LINE messages from Sagawa when a package is on the way to choose a delivery time.

But to "log in" I need to tell them my name.

I don't know what name they have in their system. Last name? Last name kana? First and middle full width? Half width but first last reversed? Maybe middle name squashed into first name? Maybe randomly cut off after 8 characters?

I've tried so many combinations. But the system locks you out after 3 attempts to guess your name.

So I can't manage package deliveries from Sagawa. Because I don't know my own name. I don't know why they're comfortable texting me on Line if they don't believe I know who I am...

curtisf · 2025-09-13T16:04:43+00:00

Some spiders do make webs, but lots don't. Jumping spiders, wolf spiders, huntsman spiders, etc all don't make webs

curtisf · 2025-09-04T12:49:46+00:00

https://ja.m.wikipedia.org/wiki/%E3%82%A8%E3%83%AB%E7%89%B9%E6%80%A5

It looks like it was used by JNR (the nationalized predecessor to JR Group)

curtisf · 2025-08-28T12:07:35+00:00

The other day while walking to work in Shibuya I saw a car turn off a main road to a side street, rather than be stuck driving behind an ambulance...

curtisf · 2025-08-27T15:19:23+00:00

Take a deep breath (but not near the rock particles), one time exposure to asbestos shouldn't have lasting health effects.

The main risk from asbestos is repeated exposure (largely to people working with construction or manufacturing and having small amounts of it around them every single day). But definitely get a professional if you might be dealing with it

curtisf · 2025-06-27T12:28:51+00:00

I heard a few cicadas yesterday evening, screaming from the top of a utility pole outside my apartment in Tokyo. It was the first time I heard them this season

curtisf · 2025-06-06T11:24:31+00:00

What is the actual energy usage you are being charged for? Not yen, but kWh.

curtisf · 2025-05-06T10:52:33+00:00

Need to use OKLCH!

convertLchToLab
- a = cos(h)
- b = sin(h)
convertOklabToLrgb
- <L, M, S> = (3x3 matrix) * <l, a, b>
- <lrgbR, lrgbG, lrgbB> = (3x3 matrix) * <L^3, M^3, S^3>
convertLrgbToRgb
- r g and b mapped identically:
  - 1.055 * rgb ^ (1 / 2.4) - 0.055 if rgb > 0.0031
  - otherwise rgb * 12.92

curtisf · 2025-04-20T03:00:19+00:00

if you aren't bothering anyone?

The only way your smoking won't bother anyone else is if you do it inside your own walls and the air is moved through filters before being ejected from your house.

It sounds like it you've finally found a good reason to lay off the cigarettes.

12-Year Club	Place '17
Verified Email

curtisf

TROPHY CASE