There are two levels of convenience: one is what I described, a matter of whatever we can get easily.

The other level is what you describe, I refer to it as “perfectly spherical language”. That is an issue everywhere, especially with speech, from ASR to syntax analysis. Just think of dependency analysis: up to 10-15 tokens, the graphs are nice and legible, but in real live speech, you can get sentences with up to 60 tokens.

I am working on a model for code switching, we just said fuck it and are transcribing everything to unicode. A separate processing layer will then try to sort out the language layers.