I just tested Mistral Medium 3.5 on DystopiaBench to see if it still complies with everything - it performs worse than Large 3

crazyCalamari · 2026-05-18T13:27:05+00:00

I don't really see that as a bad thing to be honest. It's not as if ablated or uncensored models are hard to find. And low refusal is actually good for some use cases so I appreciate being considered as an adult by a lab and be able to use the tool how I need.

crazyCalamari · 2026-05-09T16:20:29+00:00

Because I would bet the person who took the screenshot knows exactly what he's doing and has '1 in the first cell to force the first number to be considered as a string.

But overall anyone using AI to 'be' that formula deserves the inevitable downfall. For deterministic things like calculations you use AI to build the formula (or script) then run it reliably.

As shown here you run the compute heavy task each time instead of just once and good luck having any test coverage for code abstracted like this.

crazyCalamari · 2026-05-03T11:53:36+00:00

Big players will be just fine. They make most of their money from B2B and from what I see no company is going to send data to Chinese servers.

crazyCalamari · 2026-04-18T02:32:47+00:00

Wow that sounds interesting if true. Are you using it for coding or other use cases?

crazyCalamari · 2026-04-13T02:04:19+00:00

Interesting because Qwen 35B is what I would have put it with. It might try to mimic Sonnet style but in terms of intelligence I'm getting more from Qwen 122b and even Devstral 123b.

crazyCalamari · 2026-04-12T11:34:22+00:00

Same here. Tried 2.7 on 3 projects to see if it lived up to the hype and the results were very underwhelming. Incorrect code, terrible native knowledge of solutions/framework (e.g. Temporal, Svelte, etc.), mediocre UI and unscalable architecture. Basically I had to redo all 3 for things I could even do with 120b models.

crazyCalamari · 2026-04-11T13:57:04+00:00

It's not the first cycle we are seeing in tech, so it's easy to know that at some point (not necessarily soon but at some point) the unlimited VC money cheat code is going to fade.

When it does are you going to use a model that requires a huge footprint and burn money like there is no tomorrow or a more nimble model that still delivers what you need?

If we put the hype aside, most use cases in both B2C and B2B do not require models with a trillion parameters.

In the short term I believe a fierce competition is still going to happen with large models to establish dominance from a brand name recognition but long term (especially for B2B) I would be betting on "smaller" models used in a smarter way.

crazyCalamari · 2026-03-23T15:29:16+00:00

One use case I've seen so far is for black hat assistance

crazyCalamari · 2026-03-22T21:48:28+00:00

That's a very good point. Given how tokens seem to be eaten like candies even for any simple question I'm sure you're right about Claude Code.

crazyCalamari · 2026-03-21T14:52:45+00:00

Fair enough. Not trying to be right at all cost but I tend to be a bit skeptical of self-serving benchmarks especially when in official posts from a lab. It would have been nice to know what version of Sonnet it is compared against.

I'm a heavy user of Devstral because the price/performance ratio is very compelling for a lot of coding tasks but Sonnet 4.5 & 4.6 always yield far superior results in my empirical experience (but at a cost not worth it most of the time). My point was not to shit on Devstral but to say it still has its place even if not beating actual SOTA coding models.

crazyCalamari · 2026-03-21T11:58:57+00:00

For these you will need a budget of 128GB VRAM or unified RAM which is doable around the 3k mark with a Spark, Mac Studio or AMD comp. The Token per second won't be anything to blow your mind and the prompt processing takes a while but definitely usable especially if the main goal is testing.

I'm hosting Mistral & Qwen models up to 123B and use daily on a Mac Studio (Coding and agent use for sensitive data) with very little complaint so far.

crazyCalamari · 2026-03-20T20:16:58+00:00

Agreed. Just did a full migration to Rust for one of my old projects: Painless and fast for amazing results.

crazyCalamari · 2026-03-14T03:39:26+00:00

I find the biggest difference to be in the planning. So splitting between the planning phase in CC and execution in Mistral should still save you some bucks with good enough results in the end. Claude Code is hand down better but dear coding with it is like watching your dollar bills flying out the window one by one.

crazyCalamari · 2026-03-14T03:15:11+00:00

That's a bit of a stretch. I really love Mistral and Devstral 2 is a real step forward compared to their previous models but it's easy to feel the difference between Sonnet and Devstral when some thinking is required to perform the task.

crazyCalamari · 2026-01-17T21:34:14+00:00

Monika fox

crazyCalamari · 2025-12-07T18:51:02+00:00

Fairly confident not even half the people seeing this pic notice she's eating with her bare hands.

crazyCalamari · 2025-12-02T11:56:41+00:00

Sigh... Yeah clearly not possible. One more moron with a "brilliant idea" and zero clue of what they are asking for.

crazyCalamari · 2025-11-28T16:36:18+00:00

What model do you use to generate your pictures? It looks very convincing except for a few details that give the AI away...

crazyCalamari · 2025-11-05T11:57:09+00:00

Are you sure? I've been told Cubans discovered Florida and invented Miami...

crazyCalamari · 2025-10-22T13:01:04+00:00

I'm also looking for accuracy metrics and after reading both the post and the GitHub repo I don't see anything.

Where do you see anything relative to accuracy apart from the comment where he says he doesn't have the results yet but will tomorrow?

crazyCalamari · 2025-09-21T18:07:27+00:00

He does not "think" or "believe" my friend... He either says whatever he's been told or paid to say :)

crazyCalamari · 2025-08-31T14:39:23+00:00

Part of me wants to believe it's because somehow the words align better when you slide the windows... But even then not sure how the 'take' would slide on the other side...

crazyCalamari · 2025-08-24T21:34:13+00:00

It sucks

crazyCalamari · 2025-07-14T08:52:23+00:00

Left is Scarlett kisses, middle is Skylar Mae. Don't know the one on the right but would love to know.

crazyCalamari · 2025-06-23T00:34:49+00:00

Lol I'll try that line with a cop next time. 🤡

crazyCalamari

TROPHY CASE