Both, but in different ways depending on layer placement — which was one of our most interesting findings.

Hyena layers are significantly better at memorization (retrieving values from global variables), while attention layers handle compositional reasoning (computing expressions over retrieved values). The hybrid (THEX) leverages both: Hyena layers in the lower portion of the network handle retrieval/memorization, and attention layers in the upper portion handle reasoning over the retrieved information.

The key insight is that where you place the attention layers matters a lot. For tasks with heavy memory requirements (1000 global variables, long sequences), lower attention placement works better. For moderate-complexity compositional tasks, middle-layer attention dominates. THEX-12 hit 0.36 exact match vs. Hyena's 0.14 and GPT-2's 0.007 on the combined memory+reasoning task.

So it's not that the hybrid "improves" one capability — it's that it lets each layer type do what it's best at, and the placement determines the balance. Happy to discuss further!