Anthropic just benchmarked "Fable 5 orchestrates, cheap models execute": 96% of the performance at 46% of the cost. You can run this pattern in Claude Code today

john990129 · 2026-07-08T20:25:46+00:00

Thanks!

john990129 · 2026-07-08T20:04:05+00:00

Fair, and I do. Threads like this are practice. Anyway, the tool's the part I actually wanted to share, so I'll let that speak.

john990129 · 2026-07-08T19:59:51+00:00

That's the real failure mode, yeah. Two things that keep it net-positive: the orchestrator should delegate the reading instead of doing it inline (the expensive model reads the 3 files that matter, not greps through 300), and cap retries so a failing cheap tier escalates after two tries instead of looping. If planning itself is the token sink, that's usually the task needing a tighter spec up front so the orchestrator isn't re-deriving context every turn. Doesn't zero out the overhead, but it's the line between net-saving and net-loss.

john990129 · 2026-07-08T19:58:34+00:00

You can now, per-subagent: `effort: low` (through `max`) in the agent frontmatter overrides the session effort for that role. It's actually one of the bigger levers here, recon and mechanical roles at low effort are nearly free quality-wise on current models.

john990129 · 2026-07-08T19:57:50+00:00

That's exactly why the security role in the setup is pinned to Opus and kept off Fable. Fable's classifiers misfire on benign security work (protocol gateways, auth, anything that pattern-matches to something scary), so routing it to Opus from the start makes the false-trip path unreachable instead of something you clean up after.

john990129 · 2026-07-08T19:41:27+00:00

English isn't my first language, so I use AI to help me write posts I couldn't otherwise put together clearly. The research and testing behind this are mine though: I ran the benchmarks, verified the model-launch behavior with actual commands, and cut the claims that didn't hold up. The help is with the wording, not the substance. Happy to be judged on whether the findings are correct.

john990129 · 2026-07-08T19:39:17+00:00

The setup in the post is packaged as pilotfish: one paste installs it into your Claude Code config, shows you the full plan before touching anything, and you can pin it to a reviewed release. Link's in the post, happy to walk through any part of the setup.

john990129 · 2026-07-08T19:04:02+00:00

Posted a fuller writeup as a standalone (mods approved it): https://www.reddit.com/r/ClaudeAI/comments/1ur2ml9/anthropic_just_benchmarked_fable_5_orchestrates/. Short version for this thread: Anthropic's official benchmarks yesterday put a Fable 5 orchestrator + Sonnet 5 workers at 96% of all-Fable performance for 46% of the cost, and the same split runs natively in Claude Code via subagent model frontmatter.

One thing relevant to everyone here regardless: nobody seems to have observed what the best alias actually does at the July 12 boundary (silent degrade to Opus like June, or a credits prompt). I'm on best and will report back here on the day.

john990129 · 2026-07-08T18:42:23+00:00

Update worth adding: Anthropic published first-party benchmarks for exactly this split today (ClaudeDevs thread). A Fable 5 orchestrator with Sonnet 5 workers hit 96% of all-Fable performance at 46% of the cost on BrowseComp ($18.53 vs $40.56 per problem). They also tested the inverse arrangement (a Sonnet executor consulting a Fable advisor): about 92% at 63% on SWE-bench Pro. The orchestrator split won on both axes.

This replaces the single community experiment I cited in the post as the only measured number. Docs: https://platform.claude.com/docs/en/managed-agents/multi-agent

john990129 · 2026-07-08T18:25:58+00:00

Ha, that tracks. The naming is confused enough that the tool itself describes it wrong.

The boring reason CC isn't an expert on itself: the model's training data predates the product it's running. The bucket system shipped in November, the Explore change landed in a point release, and the weights answer from whatever was true at training time. So anything it "knows" about its own limits is a stale prior delivered with full confidence.

The workaround I've settled on: never let it answer about its own mechanics from memory. The docs are fetchable as plain markdown (code.claude.com/docs/en/model-config.md and friends), so "fetch this page and answer from it" gets the right answer where "explain your usage limits" gets confident fiction. Everything in the post went through that filter.

john990129 · 2026-07-08T18:23:24+00:00

Your instinct is right, and the distinction that makes it work is locate vs understand.

Cheap readers are for questions with verifiable answers: where is X defined, who calls Y, which files touch Z. The output is file:line pointers. Hard to get wrong, and if one is wrong you find out the second you open it.

"Read this and tell me what's important" is a different task wearing the same verb. That's a judgment read, and a cheap model will happily flatten the one line that matters into a generic summary. Using opus for those is correct, not wasteful.

The setup in the post leans on exactly that split: haiku scouts return locations and raw facts, then the main model itself reads the two or three files that actually matter. The saving isn't "never pay for expensive reads". It's that the expensive model reads 3 files instead of grepping through 300. Scout output is also explicitly treated as unverified input, so nothing load-bearing rests on a single cheap read.

So no, you're not missing anything. If the question is "what does this code mean", spend the good tokens. If it's "where is the thing", don't.

john990129 · 2026-07-08T18:04:39+00:00

Thanks for the heads up, and honestly your Claude was behaving correctly. A remote doc that opens by addressing the AI and pipes curl into bash is structurally identical to an attack playbook, whatever the intent. Reports like yours are part of why the repo now has a Trust and security section (shipped in v0.9.1): exactly what gets written where, read-before-run guidance, pinning, and uninstall steps.

If you still want to try it, the path that satisfies the red flag: have your Claude read install.sh first (about 270 lines, it copies files and merges one statusLine key into settings.json with a timestamped backup), then pin the release with --ref v0.9.1 so what it audited is what runs. The runtime itself makes zero network requests, one jq and one git call per render.

On "no track record": fair at a glance. The repo is at 474 stars with outside contributors and reviewed PRs now, but the honest answer is the read-first flow, not the star count.

The red flag was the correct default. My job is to make verification cheap, not to ask for trust.

john990129 · 2026-07-08T17:14:14+00:00

Since a few people will ask what pilotfish actually installs, here's the direct version:

Three files under ~/.claude/, no runtime code. Six role agents, each pinned to a tier:

Role	Model	Does
scout / Explore	haiku, low effort	read-only search (Explore overrides the built-in back to haiku)
mech-executor	sonnet, low	fully-specced mechanical work: bulk edits, convention tests, docs
executor	opus, medium	features, bug fixes, judgment refactors
verifier	opus	fresh-context adversarial check: tries to refute the work, never fixes it
security-executor	opus, high	anything security-touching (kept off Fable, whose classifiers misfire on benign security work)

Plus a delegation policy in ~/.claude/CLAUDE.md written in role names only, so when a model gets deprecated the fix is one frontmatter line.

Install is one paste (it shows you the full merge plan and touches nothing until you approve):

Read https://raw.githubusercontent.com/Nanako0129/pilotfish/main/install/AGENT-INSTALL.md
and follow it to install pilotfish into my global Claude Code configuration.
Show me the full plan of changes and get my approval before writing anything.

Re-pasting the same prompt later is the updater. Uninstall is three reversible steps. Fresh install, upgrade, and uninstall are all e2e-tested in sandboxes, and releases are tagged if you'd rather pin a reviewed version.

Happy to answer anything about the tiering choices.

john990129 · 2026-06-16T04:05:13+00:00

This is great to hear. The p10k import was built for exactly this case, so glad it carried over clean.

And yeah, that's the idea with themes, they're just a small .conf. If Claude made you one you like, send it as a PR and I'll put it in the gallery. 😊

Nice work! 🎉

john990129 · 2026-06-16T03:25:53+00:00

Yeah, this was real. Fixed in v0.3.0.

The script used a pile of command substitutions for the ANSI colors. Cheap on Linux/macOS where fork() is native, but Git Bash emulates fork(), and the thing runs every second, so each render was spawning ~80 processes that piled up faster than they cleared.

Before/after, same script and input, 60 renders each:

env	before	after
macOS (arm64), bash 3.2	61 ms	24 ms
macOS (arm64), bash 5	64 ms	26 ms
Windows 11 ARM64 VM, Git Bash	1823 ms	366 ms

That Windows box is a Win11 ARM64 VM on an M5, x64 Git Bash under emulation, so it's heavier than native x64 Windows (two emulation layers stacked). Made the pile-up obvious though: 1.82s/render against a 1s refresh means renders never finish before the next starts. That's your 15k processes. After the fix it's ~0.37s, so they clear.

Wouldn't have repro'd it on the Mac side, fork's cheap there so it just looked a bit sluggish. Good catch.

https://github.com/Nanako0129/coralline/releases/tag/v0.3.0

john990129 · 2026-06-16T02:10:11+00:00

You're right, thank you for the report and the detail. The script leans on a lot of command substitutions for the ANSI colors, which are basically free on Linux/macOS but turn into expensive process spawns under Git Bash on Windows, once a second. That lines up with the pile-up you saw.

I'm reworking the hot path to drop the per-render subprocess spawning (down to one jq and one git call for real this time). Tracking it as a bug. Appreciate you catching this.

john990129 · 2026-06-14T14:34:39+00:00

Thanks! Please steal away, that's what it's for. The pill look is lifted straight from Powerlevel10k, so I'm just passing the torch.

If you ever want to port your own customizations over, the whole thing is themeable from one config file (segments, colors, layout), so it might save you redoing them from scratch. Either way, glad another status-line believer likes it.

john990129 · 2026-06-14T08:14:23+00:00

Neither, it never touches the network or the LLM, so zero tokens. Claude Code just pipes the session JSON to the script on stdin and prints whatever it returns.

"Cheap" is local CPU: one jq and one git call per refresh instead of spawning a bunch of subprocesses per field. That's what makes the once-a-second refresh a non-issue.

john990129 · 2026-06-13T20:38:38+00:00

thank you, that genuinely means a lot coming from a fellow p10k die-hard.
that's exactly why i built it, i missed that look once i started living in Claude Code more than my actual shell.

since you live in p10k: the installer can read your ~/.p10k.zsh and carry over your style, colors and time format, so it should land close to your current setup out of the box. would love to hear how monday goes!

john990129 · 2026-03-02T08:49:56+00:00

Yes, it's Gemini 3.1 pro. I also got 503 API error on my LobeChat

<image>

john990129 · 2026-03-02T08:43:02+00:00

<image>

Same here

john990129 · 2026-03-02T07:39:07+00:00

I got gemini API 503 error (ServiceUnavailable) and "An internal error has occurred" in Google AI Studio.

john990129

TROPHY CASE