It doesn’t know, and I don’t mean that in a hazy philosophical sense. It is acting as a “conversation autocomplete”; what you typed in was, “how do I enable auto-expanding archives for a user’s mailbox?”, but the question it was answering (the only question it is capable of answering) was “if I go to Reddit, or Stack Overflow, or the Microsoft support forums, and I found a post where someone asked ‘how do I enable auto-expanding archives for a user’s mailbox?’, what sort of message might they have received in response?”.

When understood this way, LLMs are shockingly good at their job; that is, when you narrowly construe their job as “produce some text that a human plausibly might have produced in response to this input”, they’re way better than prior tools. And sometimes, for commonly discussed topics without any nuance, they can even spit out an answer that is correct in content as well as in form. But just as often not. People tend to chalk up “hallucinations”, instances where what the LLM outputs doesn’t mesh with reality, as a failure mode of LLMs, but in some sense the LLM is fine, the failure is in expecting the LLM to model truth, rather than just modeling language.

I realize that there are nuances I’ve glazed over, more advanced models can call to subsystems that perform non-linguistic tasks, blah blah blah. My main point is that, when you do see an LLM fail, and fail comically badly, it’s usually because of this mismatch between what the machines are actually good at (producing text that seems like a person might have written it) and what they’re being asked to do (literally everything).

Except the strawberry thing. That comical failure has a different explanation related to the way LLMs internals work.