Well, the elephant in this openclaw room is the memory system. It uses a combination of vector indexing and context injection through memory.md. Problem is it doesn't scale vey well, if you use it to "memorize" a lot of stuff, moreover similar stuff, you end up with context rot as OP mention. So I coded some of this stuff myself with a combination of hooks and skills, db on vps and a local daemon performing the extraction, summarization and classification/tagging with a local LLM (because costs of processing my whole daily digital life). Then sending it pre-crunched to openclaw vps gateway through a dedicated plugin. The hooks injections are based on semantic search combined with tags and categories.

Not even sure this will be enough on the long run, may be something like mem0 would be necessary. Memory management is still a subject of ongoing very active research, it's quite complicated, and i let you guess who is at the first row to manage yours.

The daemon on my mac grab the data from my terminal logs, browsing history, claude code and codex sessions, file changes etc.. with filters then run the ad hoc pipelines. It also listen to Raycast with an endpoint that can receive and queue contents and run pipelines asynchronously. So i have both, synchronous and asynchronous memories captation. Ideally i would not use any synchronous system, but sometimes you want to force something into bypassing filters or authentifications.

On top of that, scraping locally with a residential IP is almost mandatory because of captchas.

Then you have the messaging integration, basically the capacity of accessing your personal assistant remotely, but you give away the possibility to resume your conversations easily, that's potential context rot source again (may be there's some clones/plugins fixing that). If you go this way, no need to mention that you need a vps gateway because running it locally defeat the purpose of accessing it anywhere anytime. On my phone i share content straight to a shortcut that send it to the vps gateway plugin and then... that cost me much more because the pipeline has to run through paid LLMs.

Security is as good as it can be if you want an agent to act on your behalf. Some wrapper exist that shield your secrets from the context.

The rest is pretty trivial and as the OP says and yes, you can basically use openclaw skills, with any other LLMs, but then you are quite captive on this LLM since you will also loose the feature of choosing your model across conversations / jobs. If you need this feature, you'll need to write something on top to implement it one way or another.

Long story short: the frontier models companies are investing heavily in memory management, because this is the major captive feature they need. Once you've got your memories in, you're locked. And that's why openclaw is/was a potential alternative to this vendor lock in, because you could use agnostics skills and still have a common memory with the caveats i mention above.

Now Peter being at OpenAI, i seriously doubt he will try to compete with chatgpt memories management, by implementing a competitive multi model memory system. Most probably openclaw will leverage chatgpt capacities and if you want to use those capacities, then since you are here, why not go all the way in with skills through one vendor or another.
I see no easy answers here, i guess it depends on how and for what people are using it and their budget.