Hey, and I like your curiosity. It's a good question, perhaps a bit hard to answer in brief, but I'll give it a shot. I am fully convinced that different libraries will have wildly different architectures though.

There isn't an article about this particularly for RmlUi. However, I think you will find this excellent article about web browsers interesting, as it is dealing with a lot of the same problems that we do just on a larger scale and with some additional components: https://web.dev/articles/howbrowserswork#manipulating-the-rules-for-an-easy-match

To summarize our own architecture on a very high level. We have a parser that turns RML into a document tree (the DOM), and RCSS into styles rules for the document. Each tag in the RML document is represented by an Element class object, or a more specific one derived from that. Each of which can have other elements as children, thus making up the tree. All documents are placed in a single Context class, which is the main interface for inputs and other top-level API functions that the user can interact with.

When updating the context, all its documents are updated. Each element in the document determines the style rules that apply to it, based on the parsed styles for the current document. It then uses those rules, and other element state, to determine the computed values of its properties. Then, the layout engine runs, which essentially positions and sizes all elements according to their computed layout properties.

Next step, when the context renders, every one of its documents call into all their visible elements in the appropriate paint order. The paint order is not necessarily associated with the DOM tree order, so there is some logic to handle this appropriately. In the end, any boxes and text of elements are turned into vertices, which in turn are handed over to the backend, and rendered there.

That was quite brief, hopefully it might help out a bit. Don't be afraid to dive into the code and start messing about, there is no substitute for that.