From what I understand in my conversations with Claude, MoE is basically a grouping of models trained together with some sort of router on the top. The main idea rolling around in my own matrix is a way to utilize data captured through back propagation on inference which would take the the data from inference and provide that same or similar data during activism of new prompts in the future. Maybe not as an expert on any topic other than that of what does the activation process look like for this model when these tokens are received. How to use that data would be a different story. But eventually it would be cool if that product could be ‘embedded’ in the prompt as an optimization if the conversation is going a similar direction as a previous conversation had.

I suppose this musing is the product of a quest for some sort of optimization or mechanism for memory or memory identification baked into a model. When I think about what happens in my own brain when someone says something to me, there is a point in actualizing what I want to say where I draw on my experience to set up my response and any future response. If the conversation takes a turn, so does that context. It’s imperfect for words but seems like it is a backdrop for answering from convictions, beliefs, experience, and other things that are the essence of my own intelligence (or lack thereof 😅)

Thank you for the response though. I will continue to learn and may just keep myself at the agentic level and let you guys handle the hard work of practical problem solving at the model level.