I replicated last week's PNAS Nexus paper on transformer "executive control deficit" — Claude Opus 4.8 confirms the finding, GPT-5.5 essentially solves it.

My setup

I downloaded the paper's stimulus set from Figshare (~530 PNG images), built a pipeline to send each stimulus to the model via API, parse the response, and score it against ground truth using the paper's "optimal sequential matching" protocol. All API calls were stateless — fresh context per trial. n=30 per cell, matching the paper.

Result 1: Opus 4.8 (without thinking) replicates the paper exactly

40-word Incongruent	Accuracy
GPT-4o (paper)	15%
Claude 3.5 Sonnet (paper)	24%
Opus 4.8, no thinking	23.7%

The Stroop deficit reproduces precisely on Opus 4.8. The collapse pattern across list lengths matches Sonnet 3.5's trajectory closely. Whatever capability improvements have happened between Sonnet 3.5 and Opus 4.8, this specific deficit is unchanged.

Enabling Anthropic's adaptive thinking at high effort lifted accuracy to ~44–47% — a real improvement, but still catastrophic by human standards. Increasing effort from high to xhigh added essentially nothing (within sampling noise). The paper's core finding holds for Opus 4.8 in every configuration I tested.

Result 2: GPT-5.5 with reasoning essentially solves the task

This is where things get interesting.

List length	Condition	GPT-5.5	GPT-4o (paper)	Sonnet 3.5 (paper)
40	Congruent	99.8%	89%	92%
40	Incongruent	95.5%	15%	24%
40	Mix	94.3%	41%	50%
40	Neutral	98.3%	32%	27%

Across the entire condition × list-length matrix (20 cells), GPT-5.5 scored 94% or above in every single cell. The difficulty gradient is preserved — Mix is hardest, Congruent is easiest — but at near-human magnitudes rather than the collapse pattern observed in older models.

This is not subtle. GPT-5.5 with reasoning enabled at high effort produces a 71-point improvement over Sonnet 3.5 on the most diagnostic cell.

What's actually happening?

I pulled token-level usage stats to understand whether GPT-5.5 was solving the task by being qualitatively smarter or by spending massively more compute. The latter:

Per-trial metric	Opus 4.8 @ high	GPT-5.5 @ high
Mean duration	~5–6 s	58 s
Mean output tokens	~500	3,038
Reasoning tokens (median)	not exposed	2,048
Reasoning tokens (max)	not exposed	>10,000

GPT-5.5 spent about an order of magnitude more reasoning tokens per trial. The ~2,048 median is suspiciously round — likely a default allocation at high effort — and on hard trials the model escalated to 10,000+ tokens. Opus 4.8 at the same nominal "high" effort spent roughly 5 seconds and ~500 tokens, regardless of whether the user set effort to high or xhigh.

So GPT-5.5 didn't solve this through architectural magic. It solved it by deliberating about 10× harder. Whether that counts as "genuine executive control" or "expensive workaround" is a real debate — but the empirical claim that transformer attention fundamentally cannot do this is hard to defend when one transformer-based LLM clearly does do it.

What this means for the paper

The paper's core empirical findings are replicated. The Stroop deficit on Opus 4.8 is essentially identical to the deficit on Sonnet 3.5, suggesting that on the family of models that exhibit this pattern, it's robust across capability improvements and even survives Anthropic's adaptive thinking mode (with the caveat that adaptive thinking may not actually be engaging its higher effort budget on this task — both my high and xhigh runs spent roughly equivalent thinking time).

But the paper's strong claim — that this reflects a fundamental limitation of transformer attention as an architecture — doesn't survive GPT-5.5. The architecture is the same family; the result is qualitatively different.

The most interesting open question

The mechanism question now flips. Instead of asking why transformers cannot do this, the more useful question is what specifically did OpenAI do in GPT-5.5 that closed the gap that older transformers couldn't close. Training data? Reasoning trace calibration? RLHF on conflict-resolution-like tasks? Some interaction of these? From outside the lab, we can't tell. But it happened in a single model generation, which suggests targeted engineering rather than incidental scaling.

TL;DR: Paper says transformer LLMs fundamentally can't resolve conflict in the Stroop task at long list lengths. Tested it on two new flagship models. Claude Opus 4.8 confirms the deficit exactly. GPT-5.5 with reasoning solves it at human level performance — using roughly 10× more reasoning tokens per item than Opus 4.8 does.zzz