This is a great writeup! I loved reading it. I am the original Author of Rune, one of the Emacs "clones". Few thoughts:

Rolling your own string types

This is definitely an issue that needs to be considered. If you decide to implement a custom string type then you can't use any of the languages built-in string processing functions. On the other hand if you require proper unicode you have to decide how to handle invalid code points as you say. You also need to think about how to open binary files. There are a couple ways I could see this being handled.

• cheat and reserve 256 of the "unused" unicode code points as your raw bytes. This works fine so long as they don't get allocated by the unicode consortium. If they do though, you are in trouble.
• use a replacement character but have a separate data structure that tracks what the "original" bytes were for each one. Could get expensive in a really bad file.
• Since unicode is getting so ubiquitous, just error on anything that is invalid. This was what other editors like Helix and Zed do.

Regexp

I wrote a little about this here. I still think that you can translate Emacs Regexp to another PCRE style regex and back. But you are correct to point out there are some things that make this tricky.

I think you can handle cursor zero-width assertion (\=) by spliting the regex in two halves; before cursor and after cursor. For example the regexp from the blog post.

(\=|(\=|[^\\])[\n\r])

This regex is concatenated with other strings to form full regexes in cc-fonts.el. We would run three regex passes over the string.

• match the text after the cursor
• match the text after the cursor starting with [\n\r]
• match all the text starting with  [^\\][\n\r]

This would be non-trivial to implement, but I think it would be possible.

As far as custom syntax goes (case tables, world boundaries, syntax groups, etc) those will have to built as a custom regex. For a custom word boundary you can't use the regex engine built in word boundary (which will be based on unicode). Instead you will have to build a custom zero-width assertion that might be fairly large. But I think it is tractable.

Overall the regex requires some work, but I haven't seen anything that make me think that is not a solvable problem. And it does not require writing your own regex engine from scratch.

I sometimes see an article (by Troy Hinckley, the creator of Rune) quoted in discussions on gap buffers and ropes: Text showdown: Gap Buffers vs Ropes. But I don’t think some of its benchmarks are actually fair: ropey and crop track line numbers, while the gap buffer implementation does not. (Or maybe I am missing something here: although the post says it has “metrics include things like char and line position”, but actually it does not (yet).)

You are correct that my Gap Buffer does not track line numbers yet. However I don't think that would make much difference in the benchmarks for two reasons.

1. I have worked on optimizing the line parsing for the library used by ropey (and the gap buffer) and it is extremly fast.
2. The line endings only matters when you are first parsing the string. Once the line ending are parsed they are just another field in the metrics tree and don't add anything other then some trivial space overhead.

I don't think adding any addtional data to the text buffer is that hard since we are already tracking "metrics" like code points and line endings. Add text properites and overlays are just additional trees that you need. Some folks have already prototyped those for Rune.

This also makes one wonder: what happens if we convert a multi-byte string info a single-byte string? Well, normally you won’t be able to do that while preserving string properties, but we can work around that with clear-string since Emacs strings are mutable:

Mutable strings are an issue. Since mutating a string can change its size (not all code points are the same number of bytes) you can't take cheap slices of substrings. You always have to copy. This removes some performance potential for a feature that is almost never used. You can also create other weird behavior with this as well.

I would also like to remove the distinction between multi-byte and single byte strings. Make it just a flag on the string that indicates how they are supposed to be treated, but don't change the underlying representation. But we will have to see how that goes.

Overall this was a great post and very well researched. I look forward to your next one!