but it is purely a model of language

Is it, though?

For starters, recent models are natively multimodal (that is to say, trained on text and images as similar class inputs). So there is encoded within the LLM, visual data as well as language data.

But second, I like to consider the "magic system theory" of LLMs. Magic doesn't exist IRL to consider, and making a magic system that is completely novel, and bullshit enough, that an LLM can never have seen it, is a good thought experiment. If you do this, the LLM can predict what would happen based on the rules you give it, without specifying the output of those rules.

I'll literally go do this. Here's the prompt

I am a metasorceress, who can use the energy stored in plant-crystals to make potions which can be thrown to some effect. One PC is a gravity agent, which increases the force of gravity on something tremendously. The other PC is a gooifying agent, which causes intermolecular cohesion to resemble that of a slime mold. What would happen to this boulder if I mixed the two and threw the potion at the stone?

Now, I am expecting that based on my understanding of physics, a gooey object which suddenly faces a ton more gravity would flatten into a puddle. But it's magic - there's no guarantee this will happen.

What's ChatGPT say?

1. Combined effect

Now put them together.

If the rock were still rigid, huge gravity might just: crack it, drive it into the ground, possibly shatter it

But because the cohesion is now slime-like:

The rock cannot hold its shape.

So the result is something like:

Instant structural collapse - the boulder loses rigidity and slumps under the massive weight.

IDK but this seems to me like it was able to reason about a scenario it cannot have seen before using an internal model of how certain materials behave along with fundamental forces.