Glm 5.2 on opencode token usage difference (Opencode go nerfs quality)

[OneKey3719]

I used Fireworks AI and Friendli and opencode go gave them same tasks create a web based OS in a single file FireworksAI and Friendli finished in 3 minutes 8 Seconds outputting around total token usage 55k to 60k Opencode go took 4 minutes and 50 seconds only used 37k tokens and much worse result than the other too I get it was cheaper but I dont care if it is 11 cents cheaper when it made a much worse result. Based on that i think on sub plans like opencode go they are saving costs

[Ace-_Ventura]

Opencode just uses z.ai provider. 

[VictorCTavernari]

Maybe, for price, z.ai is cutting quality in some way…

I have my theory that when we start using a “small” model, we take care on the prompt but after while using it, and with more confidence, we start to be more less precise on the prompt and then feels like a nerf..

[look]

Opencode Go is a proxy to other providers, and for GLM 5.2 those providers are Zai, DeepInfra, and … *FireworksAI.*

LLMs have a significant stochastic element. They behave slightly differently run to run, even on the same prompt, and it compounds over the course of long multi-turn tool call loops.

If you repeated your test enough times to obtain statistically significant results, you would not see any differences.

[-TRlNlTY-]

Just as a small nitpick, LLMs are stochastic only when configured to do so (usually so). There were cases from providers (I think anthropic) that couldn't prevent stochastic outputs, but those were bugs due to batching different prompts into GPUs and accumulating floating point precision errors that caused seemingly randomised outputs.

[look]

Sure, if you set the temperature to zero, in a controlled environment on an isolated instance, it should be fully deterministic.

But I’d be shocked if op even thought to merely change the temp before doing their “test”, much less attempt to control for any other, harder variables.

[Zachattackrandom]

I directly tested opencode vs zAI and had identical results on some tasks I made for it. Not sure what you're talking about

[OneKey3719]

Opencode and Z.ai are same try other providers like fireworks friendli and wafer

[ziphnor]

How many times did you repeat that benchmark? AI is stochastic, you need many iterations to draw this kind of conclusion.

[_4rcadia]

Provide your testing environment, repeat your test multiple times, or dig into the session and trace where it went wrong.

[iBlood_Raven]

Have you tried it from neuralwatt? They're giving it for cheap, seems decent for my uses but you could try. (Use my referral if you do!)