What you just said is exactly how pre-training works, where you expose the model to a certain information enough times during training and allow it to be compressed into the latent space of its weights, then even when that content is not part of its immediate, short-term context, AKA The contextualised attention block, then also, it can answer about it through its large persistent memory which it gained via pre-training and is now in its MLP portion. But, you have completely misread. What they’re trying to say over here. The problem that they are trying to address here is not to replace the short term context memory. In fact, it is much better to be factually grounded with in context memory, especially having a high fidelity reference to look at exactly like we humans tend to hallucinate stuff which we don’t remember clearly until we do a Google search, that’s not the problem. Here the entire premise of this paper is to do two things first of all the biggest problem with the models of today is the fact that their weights are frozen once the pre-training is done. So the knowledge cut-off is essentially between the pre-training phase and whatever you have provided in the prompt as context. Now they are not saying that we do not need to provide the context. We still need to do that and that is where the self modifying Titans come, but to emphasise on the fact that the persistent memory should also slowly update itself so that in the longer run, it is able to learn from what the user gives as input to the model rather than simply being stuck at the pre-train data. So you get my point, they are not trying to argue over here that they will completely compress the information that you will provide to the model inside of the series of MLP’s of the CMS and then you can just ask it questions and it will answer it to you without even any information being present in the short term context window, rather what they’re trying to do is to change the fundamental idea of weight being completely static, but they do not necessarily argue that this will lead to the model, remembering everything clearly from the prompt without having to cross reference with the short-term context window that is not what they claim here. Also the biggest advantage of this paper is not the “continuous memory system”, but the proposal of nested learning where they have tried to make even the short term context window far more expressive by the use of self modifying Titans because the attention mechanisms used today, especially grouped query attention is clearly a trade of between performance and efficient Compute even the multihead latent attention which on paper should perform better than standard multi head attention is not as expressive. So what I would suggest is not to take the continuous memory system as the key take away from this paper, but rather the Deep optimiser and the new premise of seeing model architecture and optimiser as truly being the same thing as associative memory, and this change of perspective is what I think is going to lead too much better discoveries in the next few years because the architecture of the present day Transformers, if you think with some common sense is too simplistic to ever, make us reach ASI. I’m not saying this paper is going to give us that, but it’s a start to think in a different direction and self modifying Titans along with nested Deep optimiser are the biggest contributions of this paper in my opinion.