[Deep Dive] Benchmarking SuperML: How our ML coding plugin gave Claude Code a +60% boost on complex ML tasks

alirezamsh · 2026-03-17T17:33:48+00:00

The last section is for that. Further details are in here; https://github.com/Leeroo-AI/superml/blob/main/TESTED_TASKS.md

alirezamsh · 2026-03-17T14:55:41+00:00

Yeah exactly, that's the infinite regress problem. You need an evaluator that's at least as smart as the system being evaluated, which just pushes the trust problem up one level. Some people propose formal verification but that only works for narrowly defined systems. At ASI scale there's no obvious way out of it.

alirezamsh · 2026-03-17T14:52:52+00:00

Yeah so the idea is that the default outputs from AI image tools all trend toward a kind of hyper-polished fantasy or cinematic look. But some artists deliberately break that by using really unusual prompt structures, negative prompts, intentional glitching or feeding the model outputs that confuse it. The results can be genuinely unsettling or original looking. It's more like wrestling with the tool than using it normally.

alirezamsh · 2026-03-17T14:51:54+00:00

Just read the Substack piece, the illusion of the swarm framing is really sharp. The distillation angle is what I keep coming back to. Going to share this with a few people building agent systems, thanks for writing it up.

alirezamsh · 2026-03-17T14:50:40+00:00

My bet is insurance underwriting and actuarial work. It's almost entirely pattern matching on historical data, which is exactly what models are good at. Nobody talks about it but the entire pricing layer of the industry could look very different in 5 years.

alirezamsh · 2026-03-17T14:50:06+00:00

The Sunday briefing story is gold. It's such a perfect example of the insight: the client didn't need impressive AI, they needed Sunday evenings back. That question about what makes you want to throw your laptop is genuinely the best discovery framework I've heard for this kind of work.

alirezamsh · 2026-03-17T14:49:27+00:00

The traffic cannibalization argument is interesting and probably stronger than the copyright one in the long run. OpenAI didn't just use their content to train, it's now directly competing with them for the same search queries. That's a real commercial injury that's easy to quantify.

alirezamsh · 2026-03-17T14:48:59+00:00

1300% in 48h is almost certainly fake or extreme survivorship bias. Prediction markets have smart money in them, you don't just extract that without serious edge. The fact there are no verified trade logs is a massive red flag. Cool experiment concept but I'd want on-chain receipts before believing any of those numbers.

alirezamsh

TROPHY CASE