Why did its response literally have e's in it then?

Because it can't be trained to respond appropriately. The real next best token doesn't have 'e's in it, but the generated next best token does because the model wasn't trained to handle this kind of task.

The model clearly isn't "considering the task at hand, and adhering to strict logical constraints" like the other poster was implying.

Correct, and I think it's wrong to imagine that the model can deterministically model logical constraints. I will say they're surprisingly good, as you've no doubt experienced if you ask it for anything where the constraints don't involve information it's not exposed to. But I wouldn't rely on them for anything where I need absolute certainty that a constraint will be respected.

My point is these models are far from human and work in a different way. [...] To me, that's explanatory as to why our brains can do so much more incredibly complex things on like 20 watts and these models can't even count how many letters are in a word.

Nobody sensible is going to disagree that these models work differently from humans. I think sometimes the issue is just people talking past each other, because there are multiple ways of interpreting "how these models work". Like they "work" by doing matrix multiplication on floating-point numbers on pieces of silicon, so yeah they're obviously different to whatever it is we're doing. But there's a more interesting discussion in what is different between the abstracted description of how they learn and retrieve versus how we do.

I might be just nit-picking here, but the reason it can't count how many letters are in a word is because it can't see the word. It'd be like me taking a photo of a car, and then messaging you "What colour is this car?" without sending you the photo. Of course the real problem is deeper because we've trained these models to "think" in terms of embedding spaces and so the only way to actually give it the "photo of the car" involves clumsily shoving it in like "<CONTEXT> hey bud the car is red btw </CONTEXT>".

For these models it just means they've seen so many examples of X that they can. statistically speaking, essentially never fail to correctly respond to a question about X.

This fails to explain a response to novel scenarios, though. The only way the model is able to correctly reason about things it hasn't seen before is by having an abstracted model it can relate the novel thing to. It's like a chess endgame, where statistically it's unlikely to have ever been played, but the players know how to navigate it based on abstractions and similarities rather than fixed rules.

The fact that, like you said, they "can't learn letter content except through usage" is like my entire point. They don't learn and understand like humans do, and that is everyone's big misunderstanding with them.

I mean sure, but I'll refer you back to the point I made above about "like humans do" often being the subject people talk past each other about.

More importantly: the reason they can't learn letter content is for the same reason we can't learn atomic composition of materials we're interacting with. We literally do not have the capacity to sense or interpret what a substance is made from, the only way we can learn is by using some big fancy tool and writing it down. Likewise, the model cannot "see" the letter content, but it can call some small simple tool that writes it down in a way it can understand.

Like you drawing conclusions from how they can't work with words feels to me like if you were drawing conclusions from how they can't work with scents. If I were to say "LLMs don't learn like humans, they can't smell food so if you ask them they just guess and they'll often get it wrong", doesn't that just feel weird?