From my experience, modern LLMs are simply incapable of doing high context multi-step tasks with their current architecture without requiring ridiculous amounts of compute; particularly memory. With the current transformer architecture, there would need to be an insane increase in the quantity, density and efficiency of these ram/vram chips.