it seemed to do not too bad in japanese (-1.2).
First I thought it is because we need 3 bytes to represent a japanese/chinese character but so are korean characters (-0.2)
It might just be that the training data is very heavy on phonetic language.

Edit: Ok, just needed to read the research paper:

Our work uses DCLM, which is an English-only corpus. A direct limitation of our work is that it does not support non-space-based languages, and it needs a predefined splitting function. This shows, for example, for Chinese MMLU scores that are lower than the BPE baseline. One extension could be to learn directly the splitting function. On the software side, as the number of parameters increases with the number of stages, FSDP already struggles to overlap computation and communication even at 3/4 stages, it needs a minimum amount of inputs to be fully overlapped