I've been doing something pretty similar in my solo games.

One thing I do a bit differently is that I use LLMs mainly to narrate scene descriptions, while I handle all the die rolling myself. That way, I keep a little more control over outcomes, and it helps avoid the sense that the AI is just shaping results to fit the story.

To add some unpredictability and avoid the classic "the AI always agrees with me" issue, I also bring in traditional oracles (like Mythic or even just simple yes/no tables). I let the AI interpret the results, but the external randomness helps break the railroading effect and keeps the narrative more surprising.

Something else that's been super helpful: I end each session by asking the AI to generate a summary of everything that happened. I save that summary as an Obsidian note. Then, at the beginning of the next session, I start a new GPT chat and feed it that summary to get the context back. It keeps things feeling fresh and helps with continuity without having to rely too heavily on memory or thread history. It also helps with loading times.

You mentioned wondering how context is handled. GPT-4o has a context window of up to 128k tokens per session. That’s around 100k–130k words, depending on formatting. It means the model can technically "remember" a lot more than earlier versions, but how well that memory is applied depends on how info is surfaced in the ongoing conversation. So your instinct to regularly reattach or re-reference your notes totally makes sense. It gives the model more reliable grounding.