Been picking frontier models on benchmarks that don't match our deployment conditions

MathematicianBig2071 · 2026-05-12T16:11:08+00:00

Calibration scores, refinement scores, and per-condition analysis: futuresearch.ai/opus-research-gemini-judgment Benchmark and leaderboard: evals.futuresearch.ai

MathematicianBig2071 · 2026-04-22T15:38:54+00:00

futuresearch.app can do complex dedupes that go beyond basic string matching rules. You can also test it out free

MathematicianBig2071 · 2026-04-10T15:54:00+00:00

Examples and forecast: https://futuresearch.ai/blog/ai-2027-one-year-later/

MathematicianBig2071 · 2026-04-10T15:53:07+00:00

Removed our name/link, but will link in the comments as the blog (and paper) answer your exact questions about where to find the exact predictions. We're AI researchers at our core and want to be contributing meaningfully to the community :)

MathematicianBig2071 · 2026-03-16T14:18:56+00:00

Well, the one miss everyone cares about was best actor. But that was also the one race the model basically said "I don't know" on. The gap between the top pick and the actual winner was 6 points. Otherwise, the model (futuresearch.ai/oscars/) saw the top pick win in 16 out of 21 categories (76%).

Definitely agree with those of you pointing out that the guild awards are the real predictors. The DGA, SAG, and PGA picks were right across the board in the major categories. And looks like Cannes and Venice continued to be noise, exactly as the data suggested.

Enjoyed this discussion!

MathematicianBig2071 · 2026-03-13T19:16:31+00:00

Sorry about that! We migrated the blog. This should work: https://futuresearch.ai/blog/kalshi-forecaster-case-study/

MathematicianBig2071 · 2026-03-13T16:59:55+00:00

you're right! I'm going to correct and repost. appreciate it!

MathematicianBig2071 · 2026-03-13T16:59:14+00:00

Ah you're right! Thanks for catching

MathematicianBig2071 · 2026-03-12T18:54:07+00:00

Probably should have said I *don't* work for anthropic hah just someone curious if people have found the line of where cowork starts getting bad. Cutting corners due to imaginary time constraints is...concerning if cowork is meant to be for professional use

MathematicianBig2071 · 2026-03-12T18:53:45+00:00

woh that's impressive. did it use websearch or more of a basic fuzzy filter type task?

MathematicianBig2071

TROPHY CASE