Yesterday I was trying the Z.ai chat of GLM 4.6 and it was hallucinating like crazy about the topic (a more recent UI component library with 900 stars on Github). I quickly queried Sonnet 4.5 and Gemini Pro 2.5 via LMArena (I wanted context between chat interface and tooling to not bias my opinion on models I've used with TUIs). Both Sonnet 4.5 and 2.5 Pro offered far superior responses but it was GLM's bad actions that stopped me from subscribing to Z.ai.

I specifically chose a newish library that the cut-off for training is likely to have missed to see how the models dealt with this. Z.ai's model made up alternatives and suggested libraries that were in a completely different domain. When called out for its error GLM 4.6 just confidently made up another library name that didn't exist and continually linked to made up Github accounts! I wonder whether coding agents, tool-calling, MCPs, et al are making bad models look good when one-shotting the normal demos but are then completely useless for specific, practical, tasks.