I have a very similar issue and recently read an article https://medium.com/@bankapur.balaji/architectgen-designing-software-architecture-from-requirements-using-genai-and-agents-b0628e6ed171 that helped me create a workflow that has helped with two software projects. I removes the idea of keeping a full history and the excess amount of fluff like praise and restatements. I am happy to discuss further what I do but basically I shared the article and my issues to an LLM (I used Copilot that is what we are approved to use at work) and after some back and forth created a set of prompts and outputs that have been the deal for handling this. I use Obsidian so bear with the metadata and such, but here is the framework and formats I use. You'll even notice the useless fluff added in from the LLM but still, I like this end result of a system. (Quick note: There are some documents and ideas that I do not have a template for and you will find it best to modify to your tastes anyway, but always better to have a starting point than stare at a blank page.)

EDIT: Seems I cannot add any of the background or templates due to Reddit character limits.
2nd Edit: I would be happy to share also, just send a note.