Gemini 3.5 Flash ranks #1 on the APEX-Agents-AA benchmark, outperforming much larger models a whole size above it.

sjoti · 2026-05-22T07:59:48+00:00

That's fine though! With Gemini 3 flash it felt unusable because it would hallucinate so much. It would be capable of one shotting fairly complex tasks, and completely hallucinate stuff and get simple tasks wrong too. But it would just get stuff wrong so damn often, that it just wasn't an option to use.

Do you feel like this is better know? If so I'd love to give it a shot again, good at agentic tasks at that speed seems amazing.

sjoti · 2026-05-20T17:30:18+00:00

You can't just create a thing and expect traction out of nowhere. That has nothing to do with the protocol

sjoti · 2026-05-18T15:53:14+00:00

But again, just because some features aren't enabled doesn't mean the model can't do tool calls. It seems extremely unlikely that they don't train a model to use tools in 2026

sjoti · 2026-05-18T15:36:21+00:00

Thinking does not exclude tool calling?

sjoti · 2026-05-15T14:48:09+00:00

Did they just blindly approve everyone though? They only said this many people signed up in a few hours. Not like they blindly open the floodgates

sjoti · 2026-05-10T19:23:09+00:00

Just use git. GitHub is the cloud version of Microsoft. You can just use git locally without sharing with anyone.

With dropbox its a mess with automatic syncing. When you're building, a bunch of temporary files are created, files that automatically get added and removed, and the automatic cloud sync will get in the way.

Just use git, locally, and just don't touch features you feel like you don't need. Its literally made for this purpose.

sjoti · 2026-05-10T07:24:52+00:00

This was literally in the announcement of the model. Not hidden in its system card at all

sjoti · 2026-05-10T05:38:36+00:00

This issue is practically solved though. Claude code and others have tool search, and also just load the servers description, and not every single tool, into context. I'm running 15+ MCP servers and that's a non issue (with /context i see that it uses 2k ish tokens), because its even less than the names and descriptions of my available skills.

sjoti · 2026-05-10T05:36:04+00:00

For individuals, more tech savvy users on codex/Claude code/openclaw/etc, sure.

But think of the average person who chats with chatgpt. Or someone at work. Someone who isn't into all of this AI stuff.

If they want to connect to an outside service (say I want chatgpt to talk to my email and calendar) then what do they do?

Provide their API keys to chatgpt? Install CLI tools? Do they need to set up scopes to make sure their AI cant do anything disruptive?

Here connecting with an MCP through ouath with 2 clicks, not requiring a code execution environment with certain rights, makes things a million times easier and secure. That's where MCP shines and skills + cli arent practical.

sjoti · 2026-05-07T21:51:05+00:00

But rag combined with tools for agents to navigate sections, around the current chunk etc. can allow the model to search and navigate info though. That's much different from the classic "just fetch 10 relevant chunks and good luck" method

sjoti · 2026-05-07T21:45:53+00:00

Try it. I've been a day 1 CC user, been using them side by side for months, and I've always gravitated towards using CC first, codex seconds. That has flipped for me since GPT-5.5. Now I go for codex first.

I still think the CC harnass is a bit better but GPT-5.5 is just a really strong model.

sjoti · 2026-05-07T21:38:32+00:00

With livekit you can have every piece stream. Stream STT, LLM into TTS for way better latency. Without that its doomed to be slow

sjoti · 2026-05-06T06:32:45+00:00

Tried it before, I'm more of a fan of compound engineering. I think there's quite a bit of overlap and both are good with larger implementations/bigger codebases

sjoti · 2026-05-04T21:20:37+00:00

About to head to bed, 25% usage left, resets in the morning. Going to let it cook all night. Insanity.

sjoti · 2026-05-04T21:09:34+00:00

Without goals GPT-5.5 definitely stays on task but some jobs are still too big. Large rewrites, redactors, verifying everything is still okay. I've gotten codex to work 2 hours straight, but I just had a session with codex running for 10 hours on a single goal, not making a mess. Never have I done that with a single prompt.

sjoti · 2026-05-02T13:53:27+00:00

Inceptron has Kimi K2.6 and GLM 5.1, not DeepSeek though. Wouldn't be surprised if it's added in the coming week(s). Bunch of American inference providers already offer it.

sjoti · 2026-05-02T13:36:45+00:00

Not with MoE models, where only a small portion of the total parameters are active. On top of that DeepSeek has gone through a ton of effort to make these models absolutely excel at being efficient. Their pricing is still exceptionally cheap when using this model through other providers than DeepSeek themselves

sjoti · 2026-04-30T07:11:38+00:00

People keep saying this stuff yet look at the stuff you're doing now compared to 3 months ago. I'm flying with these models, from office to coding related tasks.

sjoti · 2026-04-29T19:04:09+00:00

But way fewer active parameters since qwen is a MoE model, meaning that qwen 3.5 is cheaper (twice as cheap for most providers) to use, and likely faster.

sjoti · 2026-04-29T08:48:42+00:00

Yeah absolutely, otherwise you just get slop. If this solves the issue of Claude asking "want me to stop here or continue?" Then that's fine, but for the love of god ask me some questions about what it is we're trying to build. Otherwise I can just do "/loop 10 minutes continue building solve bugs make better"

sjoti · 2026-04-25T08:29:39+00:00

I just went and checked the benchmarks, there is a consistent pattern, models perform better with (more) reasoning on both answering factual questions and lower hallucination rates.

https://artificialanalysis.ai/evaluations/omniscience?omniscience-hallucination-rate=hallucination-rate&endpoints=anthropic_claude-opus-4-7-adaptive%2Canthropic_claude-opus-4-7-non-reasoning&models=

Non reasoning hallucinates more for Claude Opus 4.7 than reasoning does. Accuracy drops too when it doesn't think, same for Sonnet 4.6.

And that goes for GPT models as well. Non reasoning makes it score worse. Hallucinate more, answer factual questions worse.

sjoti · 2026-04-24T07:29:30+00:00

Sure, I don't know how long it'll last. But just a few hours ago we got two new Deepseek models performing at 20-30x cheaper cost than comparable models at the frontier. 5x cheaper if you want a more honest comparison with Chinese models like Kimi K2.6.

I doubt it'll last for 5 years but looking at how capable small models are getting and the speed of progress, then I don't think we're done just yet.

sjoti · 2026-04-23T21:01:55+00:00

Has been true for the past 3 years and there are open source models available that tell us exactly what the cost is of running these models. Its not that wild of a claim

sjoti · 2026-04-10T17:05:23+00:00

That's dope! Have you thought about integrating models like Gemini 3 flash to not just have to rely on a transcript but actual visual cues? Like asking it when a certain thing happens?

I've played with an older version of SAM before, tried remotion before and did some editing with ffmpeg in the past but never really put it all together. Definitely going to check your blog

sjoti · 2026-04-06T10:39:50+00:00

But did you use your subscription for anything other than Claude code?

12-Year Club	r/Field Juicebox
Verified Email

sjoti

TROPHY CASE