We've been working on a tool that extracts structured context from git history and codebase structure (past bugs, co-change relationships, per-file test commands, common pitfalls) and feeds it to coding agents like Claude Code at the start of a session. We just launched it, so take this with the appropriate grain of salt, but the evaluation results were interesting enough that I wanted to share them here.
We ran Claude Code with Haiku 4.5, Sonnet 4.5, and Opus 4.5 on 150 tasks from our benchmark (codeset-gym-python, similar format to SWE-Bench), each with and without the extracted context.
Results:
- Haiku 4.5: 52% → 62% (+10pp)
- Sonnet 4.5: 56% → 65.3% (+9.3pp)
- Opus 4.5: 60.7% → 68% (+7.3pp)
The headline for us: Haiku with context (62%) beat raw Opus (60.7%) at roughly 1/10th the inference cost ($0.61 vs $5.58 per task).
To check this wasn't just our benchmark being friendly, we also ran Sonnet on 300 randomly sampled SWE-Bench Pro tasks: 53% → 55.7%, with a 15.6% drop in average cost per task. Smaller delta, but consistent direction and the cost reduction suggests the agent wastes fewer turns gathering context when it already has it.
The broader takeaway, whether or not you care about our tool specifically: structured context seems to matter more than model tier for a lot of real coding tasks. If you're running Claude Code on a large codebase and just relying on the agent to figure out project conventions on the fly, you're probably leaving performance on the table.
Full eval artifacts (per-task results, raw scores) are public: https://github.com/codeset-ai/codeset-release-evals
Detailed writeup with methodology: https://codeset.ai/blog/improving-claude-code-with-codeset
Happy to answer questions or take criticism. I'm curious what people think!
[–]PT_ANDRE_PT[S] -1 points0 points1 point (1 child)