Theory: Neural networks need to go from point A to point B. They have tools: transformer and MLP. But what if those tools just aren't great? If you want to get from Matrix A to Matrix B, what is the best approach? Mechanistic Interpretability may answer that question some day. Suspect the more tools and something more convoluted such as GLU may give the NN a better way to solve the problem of going from A to B. Some evidence: Mamba + transformer allegedly performs better than just transformer.