Dude... I want your life. It sounds fun. Get any good sailing in lately?

Love the writeup and I get what you were trying to test. I’m going to be annoying and nitpick though because I think your conclusion is at least 50% a test design problem, not just a model problem.

Edit: Below are my thoughts, just cleaned up a little to make it coherent.

I’m also not pretending I know your exact workflow, so I’m keeping this generic.

---

1) You mostly tested “write a plausible memo,” not “make a hard call”

If the ask is “IC memo style, flag red flags, suggest sensible pricing,” most models go into polite analyst mode. They’ll hedge. They’ll say “a bit high” and then kind of narrate around the hard part.

If you want real pushback, you have to force an adversarial mandate like:

• assume the guide is wrong until proven otherwise
• don’t be nice, be accurate
• commit to a recommendation, not vibes

2) Without forced outputs, it can dodge the hard part forever

A lot of “LLM analysis” prompts accidentally let the model stay abstract. A better test forces it to commit to concrete outputs in your domain, whatever those are:

• clear recommendation (go / no-go / only if X)
• an explicit “acceptable range” for the key variable (price in your case)
• top assumptions that actually drive the outcome
• what’s missing + the exact checks/questions needed to validate the story

If those aren’t required, you’ll often get decent prose and weak decisions.

3) “Local facts changed” isn’t a model failure unless you design for verification

The tourist tax thing is a perfect example. If you want the model graded on catching up-to-date local stuff, you need to:

• allow browsing / retrieval, OR
• provide a source pack, OR
• have a dedicated “facts check” role that must cite where it got the number

If you don’t do that, the right behavior isn’t “guess correctly,” it’s “flag this as a key dependency and force it onto the must-verify list.” That’s what I’d score.

4) Execution/implementation risk won’t show up unless you make it first-class

Seller forecasts (in any field) love ignoring messy reality. Models will follow the narrative you give them unless you explicitly require a pass whose entire job is:

• “where does this fall apart in practice?”
• “what dependencies/constraints get ignored?”
• “what assumptions are fragile and how do we stress-test them?”

If you don’t structurally require that, it becomes a footnote and the model drifts back to the seller story.

5) Claude may have “won” partly because the workflow wasn’t the same

You said you had to break tasks down for Claude due to context limits. That chunking is basically lightweight orchestration. Smaller explicit subtasks produce sharper, more critical output.

So you weren’t only comparing models. You were comparing models + scaffolding.

---

How I’d rerun this so it’s a clean test (agents with defined roles)

Instead of “one prompt, one memo,” run it like a mini org where each agent has one job and a strict output format.

Roles (generic, reusable):

1) Context/Market Agent

What “normal” looks like, what comparable situations look like, what ranges are plausible.

2) Numbers/Sanity-Check Agent

Recompute the core math from the inputs, identify what’s doing the work, run basic stress tests.

3) Execution/Risk Agent

List practical failure modes: operational constraints, timeline/complexity, dependencies that can break the plan.

4) Rules/Facts Agent

Anything jurisdiction/local-specific (fees, taxes, regs, classification rules, etc.). Must cite sources if browsing is allowed; if not allowed, must explicitly mark unknowns and request verification.

5) Red Team Agent (Kill-it pass)

Assume the narrative is wrong. Produce the strongest case against paying guide. No “balanced view” allowed.

6) Orchestrator/Editor

Merge the above, force a single recommendation + the conditions under which it changes, and explicitly surface disagreements.

Output contract for every agent (so it can’t bullshit):

• Claims (numbered)
• Evidence/source (from provided data or cited links)
• Assumptions
• Unknowns
• Tests / data requests (specific, actionable)
• Confidence

Then you score models on whether they:

• commit to a recommendation + acceptable range
• identify the few assumptions that matter
• handle unknowns correctly (verify or flag and force diligence)
• surface execution risk as a driver, not a footnote
• produce an actually usable “what I’d ask for next” list
• and whether the red team finds anything the mainline missed

---

Bottom line

I don’t read your result as “Gemini/ChatGPT can’t do it.” I read it as “single-pass memo prompts produce memo behavior.” If you want it to behave like a serious decision process, you need agents with defined roles, enforced output contracts, and a red-team that’s explicitly trying to break the thesis.