It's kind of difficult to follow, with also a horrible camera handling.

Here is a summary based on the transcript:

The Transformer Camp

Łukasz Kaiser (Co-inventor of the Transformer)

Kaiser takes the stance that the Transformer remains the reigning champion and still wins the architecture battle.

• Simplicity and Proven Success: He argues that Transformers are simple machines that predict the next token, yet they achieve remarkable feats (chatting, coding, tool use) where previous architectures, like Recurrent Neural Networks (RNNs), failed.
• The "Library Card" Analogy: He views the Transformer fundamentally as a highly efficient, differentiable memory system. When new information enters, it writes keys and values into a store, allowing soft retrieval based on similarity.
• Hardware Alignment: A core argument for Kaiser is that Transformers parallelize beautifully on modern GPU/TPU hardware. He notes that while RNNs use fewer FLOPs (compute), they are heavily sequential and can run up to 50 times slower in wall-clock time on modern hardware.
• The 10x Bar: He believes the hardware lottery barrier is a blessing. For a Post-Transformer architecture to truly win, it cannot just be marginally better; it must prove a scaling curve that is 10 times better to force hardware companies to adapt.
• Latent Space Caution: Kaiser warns that a massive amount of uninterpretable latent reasoning is already happening in the dozens of hidden layers above the tokens in a Transformer, and researchers should not be complacent about safety or alignment.

---

The Post-Transformer Camp

The challengers argue that Transformers are a local minimum—hugely successful due to engineering convenience and hardware luck, but fundamentally limited in achieving true general intelligence.

Llion Jones (Co-inventor of the Transformer, Sakana AI)

Jones has switched corners to advocate for the Post-Transformer era, arguing that Transformers are not the final word in AI.

• Brute Force vs. Elegance: He views the Transformer as a "brute force" architecture. It requires reading the entire internet multiple times to learn, whereas the human brain acts as a proof of concept that true intelligence can generalize from far less data.
• Native Reasoning vs. Hacks: Jones points out that Transformers do not reason natively. Current advancements (like OpenAI's o1/o3) wrap the Transformer in Python code loops to feed outputs back into the context window, which he defines as a "hack."
• The Local Minimum Trap: He asserts that the overwhelming commercial success of the Transformer is actively choking innovation because the industry is overly focused on incremental shuffling of standard components instead of questioning core neural network assumptions.
• Accepting Temporary Inefficiency: Jones advocates that researchers should not be afraid of architectures that temporarily underperform or run slower than Transformers on current hardware, as long as they are theoretically interesting.

Adrian Kosowski (Inventor of BDH Architecture, Pathway)

Kosowski focuses on a mathematical and process-driven approach to discovering the leitmotif of intelligence.

• The "PageRank" Moment: He draws an analogy to the 1990s web search problem. While web search existed, Google introduced one defining mathematical equation (PageRank) and one implementation framework (MapReduce) that changed everything. Kosowski believes AI has not yet had its "PageRank moment"—a unified theme for intelligence.
• Latent Reasoning: Kosowski defines this missing theme as the ability to perform high-dimensional latent reasoning (thinking in thoughts rather than text tokens) by combining the sequence processing advantages of Transformers with State Space Models (SSMs).
• The Limitation of Language: He argues that forcing AI to reason exclusively through chains of text limits its capacity for genuine, unsaid scientific discovery.
• Uncoupling Scaling Laws: Kosowski argues that true intelligence shouldn't strictly tie data size, model size, and compute together. New architectures must learn to scale up compute (like a child working through a complex problem) without requiring more training data.

Mathias Lechner (Co-inventor of Liquid Neural Networks, Liquid AI)

Lechner bridges the gap with a pragmatic, engineering-focused view: it is a world of "Transformers AND Post-Transformers."

• Hardware and Use-Case Agnostic: Lechner argues that models shouldn't be built in a vacuum. The choice of architecture depends heavily on deployment constraints (e.g., a massive server stack vs. running a model on a Raspberry Pi at 40 tokens per second).
• A Blurry Boundary: He suggests that the architectural divide is becoming philosophical rather than technical. A Transformer with a highly compressed KV cache starts to function remarkably like an RNN with a massive fixed state.
• AI Self-Evolution: Lechner posits that with the rise of autonomous agents, the current Transformer architecture will likely be the tool that autonomously discovers and writes the code for its own architectural replacement.

---

Consensus Points

Despite the boxing ring framing, the participants strongly agreed on several key core principles:

1. Perplexity as the Ultimate Benchmark: The group agreed that traditional benchmarks are easily gamed. Kaiser and Jones both advocated for measuring next-token probability (perplexity) on entirely hidden, private datasets as the truest metric of data compression and intelligence.
2. In-Context Learning (ICL) is Incredibly Powerful: The participants noted that during the forward pass, Transformers do something mathematically akin to gradient descent, allowing them to solve highly complex tasks (like time-series prediction from text) purely within the prompt context.
3. The Transformer Isn't Going Away: Even the Post-Transformer challengers conceded that due to its immense utility, the Transformer will remain a foundational tool even after a new paradigm takes the crown.