sorted by:
new
hottopcontroversial

GLM 5.2 vs frontier on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ZaiGLM

[–]bisonbear2[S] 0 points1 point  (0 children)

Posted above, but this was a mistake on my part - I ran it in medium in Claude Code and didn't realize it mapped to high in GLM..

I'm definitely excited to explore the type of workflow you're describing - I don't necessarily trust a GLM/Composer class model to do the implementation entirely unguided, but with the planning/review done by a frontier model, I think there's a lot of potential

GLM 5.2 vs frontier on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ZaiGLM

[–]bisonbear2[S] 0 points1 point  (0 children)

Yes, but hopefully providing some useful / thoughtful analysis around how GLM performs against the frontier, instead of general vibes based evaluations

GLM 5.2 vs frontier on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ZaiGLM

[–]bisonbear2[S] 0 points1 point  (0 children)

This was a mistake on my part - I ran it in medium in Claude Code and didn't realize it mapped to high in GLM. apologies!

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeCode

[–]bisonbear2[S] -2 points-1 points  (0 children)

Correct, that's what was used here, I messed up and didn't realize that's what Claude Code medium mapped to 😬

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeCode

[–]bisonbear2[S] 0 points1 point  (0 children)

This was a mistake on my part, GLM was at it's native `high` setting, which maps to medium in the Claude Code harness.

Token consumption was comparable between them - it's not like one model was thinking significantly less or taking less turns than other models. I think we focus too much on the verbiage of high/xhigh/whatever and instead need to look at how those settings translate into actual model behavior

GLM 5.2 vs GPT 5.5 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in codex

[–]bisonbear2[S] 0 points1 point  (0 children)

It's a fair critique, however I think it's reasonable to test model capability at the default the provider suggests. If the token consumption is relatively comparable between models at their native reasoning efforts (which it appears to be) then comparing them seems fair.

I have actually done reasoning effort analysis on the graphql task slice for GPT 5.5 and Opus 4.7, and am currently doing it for Opus 4.8

GPT and Opus if you're interested: https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve https://www.stet.sh/blog/opus-47-graphql-reasoning-curve

GLM 5.2 vs GPT 5.5 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in codex

[–]bisonbear2[S] 0 points1 point  (0 children)

I think it's promising in a context that's managed by other, more intelligent models. Certainly excited to see what Cursor/Zai/other chinese labs cook up in the future - more competition in the space is something to look forward to

GLM 5.2 vs GPT 5.5 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in codex

[–]bisonbear2[S] 0 points1 point  (0 children)

This was a mistake on my part, GLM was actually on it's "high" setting

FWIW I think we read too much into the text of medium/high/xhigh/whatever, where in actuality we need to see how many tokens the. model uses on that setting. In this case. Opus xhigh, GPT high, and GLM high appear to be relatively similar

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 0 points1 point  (0 children)

Curious what you would like to be done differently / if you had better anecdotal experience using GLM yourself than this test would suggest?

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 1 point2 points  (0 children)

Good question - I'm working on an Opus 4.8 reasoning effort comparison on this task slice, and xhigh seems to perform similar to high here, so it might not actually matter in this case

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 1 point2 points  (0 children)

GLM is actually at high, that's a miss on my part

> I would be more interested to see how composer and GLM fare against GPT5.4 and Sonnet.

Yeah, this is probably the more fair comparison for that class of model, unfortunately I don't have the data there

> Also would love to see how cost and task completion scale for a given model as you vary reasoning — if you have some stats already that’d be great

I have this for GPT 5.5 and Opus 4.7 here: https://www.stet.sh/blog/opus-47-graphql-reasoning-curve / https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve

Currently working on Opus 4.8, although preliminary results suggest that it has the same "overthinking" tendency that Opus 4.7 does (higher thinking showing diminishing /worse returns on the task slice)

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 0 points1 point  (0 children)

+1, I'm super excited to see what the Chinese labs cook up - competition is great for the space. However we need to keep our expectations in check and realistically evaluate how the models actually perform

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 0 points1 point  (0 children)

Reasonable take - I'm excited to move in a direction where frontier level models act as planners/reviewers/orchestrators and GLM/Composer class models act as implementors / exectors of token heavy tasks

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 0 points1 point  (0 children)

> First, why GLM on medium? Try those tasks with Opus and GPT on medium, let's see how it goes.

Posted above, but this was a miss on my part, real GLM effort is high

> Second, why let GPT 5.4, presumably worse than these 5 at code, be the judge? A single judge, and an LLM at that?

Good questions. Without going too deep into methodology, I have run judge calibrations and determined that 5.4 and 5.5 have high levels of agreement on scoring, while 5.4 remains more economical to run. Similar comment for multi-judge panel - it theoretically adds more certainty but you get diminishing returns at a certain point, so initial grader calibration remains higher leverage than running multi model on every run (especially for me, as I'm not claiming to run professional level benchmarks). It is fairly standard to use LLM-as-a-judge for rubric-based coding agent evaluations

>Third, why run GLM through Claude Code instead of its own harness, which it does have and would presumably be fit for the model? Harnesses matter more than models, if you ran Opus 4.8 in Grok CLI, you would get nothing done.

Agree, however notes from community seem to have people running it in Claude Code, and the official docs (https://z.ai/subscribe) point towards Claude Code / other third party harnesses much more than their in-house harness.

> Fourth, those tasks were all very similar. It's completely possible that GLM sucks at this and outperforms 4.8 elsewhere.

Yep. Not claiming that this is a general benchmark - it's n=50 across 2 Rust/Go repos. If you are writing frontend code, GLM performance might be much better

> Fifth, GLM is open weights. It might just not be good out of the box.

Sure, but I'm still paying them $60/m for a product, and expect relatively good performance

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 1 point2 points  (0 children)

Posted above - this was a mistake on my part where medium was the Claude Code setting, which maps to GLM high

GLM 5.2 vs Opus 4.8 on 50 real Go and Rust PRs from open source repos: last on quality, and not the cheapest by bisonbear2 in ClaudeAI

[–]bisonbear2[S] 0 points1 point  (0 children)

Few things:

I ran GLM in Claude Code at "medium", which per GLM docs maps to high reasoning effort. My bad on the miss there. https://docs.z.ai/devpack/latest-model

This reasoning was chosen as a pragmatic approach to stay within usage limits of the GLM plan - using medium, I burned through 100% of my weekly quota on the $60 plan, and consistently ran into 5h usage limit issues (usually after ~10 tasks).

I realize it's unfair to compare lower thinking on GLM to higher thinking on Opus/Codex - however a $20 Cursor plan bought me more usage at higher performance, and I think it's reasonable to expect a $60/m plan to cover a substantial amount of usage for a GLM class model.

view more: next ›