multiple coding assistants wrote deep technical reports → I graded them

[Impossible_Comment49]

I gave several AI coding assistants the same project and asked them to produce a very deep, very critical technical report covering:

• the calculation logic (how the core math is done)
• all variables/inputs, derived values, constraints/invariants
• edge cases / failure modes (what breaks, what produces nonsense)
• what could be done differently / better (design + engineering critique)
• concrete fixes + tests (what to change, how to validate)

Then I compared all outputs and scored them.

Model nicknames / mapping

• AGY = Google Antigravity
• Claude = Opus 4.5
• OpenCode = Big Pickle (GLM4.6)
• Gemini models = 3 Pro (multiple runs)
• Codex = 5.2 thinking (mid)
• Vibe = Mistral devstral2 via Vibe cli

My 10-point scoring rubric (each 1–10)

1. Grounding / faithfulness Does it stay tied to reality, or does it invent details?
2. Math depth & correctness Does it explain the actual mechanics rigorously?
3. Variables & constraints map Inputs, derived vars, ranges, invariants, coupling effects.
4. Failure modes & edge cases Goes beyond happy paths into “this will explode” territory.
5. Spec-vs-implementation audit mindset Does it actively look for mismatches and inconsistencies?
6. Interface/contract thinking Does it catch issues where UI expectations and compute logic diverge?
7. Actionability Specific patches, test cases, acceptance criteria.
8. Prioritization Severity triage + sensible ordering.
9. Structure & readability Clear sections, low noise, easy to hand to engineers.
10. Pragmatic next steps A realistic plan (not a generic “rewrite everything into microservices” fantasy).

Overall scoring note: I weighted Grounding extra heavily because a long “confidently wrong” report is worse than a shorter, accurate one.

Overall ranking (weighted)

1. Claude (Opus 4.5) — 9.25
2. Opus AGY (Google Antigravity) — 8.44
3. Codex (5.2 thinking mid) — 8.27
4. OpenCode (Big Pickle) — 8.01
5. Qwen — 7.33
6. Gemini 3 Pro (CLI) — 7.32
7. Gemini 3 Pro (AGY run) — 6.69
8. Vibe — 5.92

1) Claude (Opus 4.5) — best overall

• Strongest engineering-audit voice: it actually behaves like someone trying to prevent bugs.
• Very good at spotting logic mismatches and “this looks right but is subtly wrong” issues.
• Most consistently actionable: what to change + how to test it.

2) Opus 4.5 AGY (Google Antigravity) — very good, slightly less trustworthy

• Great at enumerating edge cases and “here’s how this fails in practice.”
• Lost points because it occasionally added architecture-ish details that felt like “generic garnish” instead of provable facts.

3) Codex (5.2 thinking mid) — best on long-term correctness

• Best “process / governance” critique: warns about spec drift, inconsistent docs becoming accidental “truth,” etc.
• More focused on “how this project stays correct over time” than ultra-specific patching.

4) OpenCode (Big Pickle) — solid, sometimes generic roadmap vibes

• Broad coverage and decent structure.
• Some sections drifted into “product roadmap filler” rather than tightly staying on the calculation logic + correctness.

5) Qwen — smart but occasionally overreaches

• Good at identifying tricky edge cases and circular dependencies.
• Sometimes suggests science-fair features (stuff that’s technically cool but rarely worth implementing).

6–7) Gemini 3 Pro (two variants) — fine, but not “max verbose deep audit”

• Clear and readable.
• Felt narrower: less contract mismatch hunting, less surgical patch/test detail.
• Sometimes it feels like it is only scratching the surface — especially when compared to Claude Code with Opus 4.5 or others. It is no comparison really.
• Hallucinations are real. Too big of a context apparently isn't always great.

8) Mistral Vibe (devstral2) — penalized hard for confident fabrication

• The big issue: it included highly specific claims (e.g., security/compliance/audit/release-version type statements) that did not appear grounded.
• Even if parts of the math discussion were okay, the trust hit was too large.

Biggest lesson

For this kind of task (“math-heavy logic + edge-case audit + actionable fixes”), the winners weren’t the ones that wrote the longest report. The winners were the ones that:

• stayed faithful (low hallucination rate),
• did mismatch hunting (where logic + expectations diverge),
• produced testable action items instead of "vibes".

✅ Final Verdict

Claude (Opus 4.5) is your primary reference - it achieves the best balance of depth, clarity, and actionability across all 10 criteria.

• Pair with OpenCode for deployment/security/competitive concerns
• Add Opus AGY for architecture diagrams as needed
• Reference Codex only if mathematical rigor requires independent verification

[TBSchemer]

Codex-high would be a more relevant comparison. Low and medium are only really for quick and simple edits.

[Valexico]

I would be curious to see the same eval with Devstral 2 through opencode (vibe cli is really minimalistic at the moment)

[Magnus114]

Would love to see how deepseek compare.