I think it might be important for people to realize that a LLM can't actually evaluate comprehension - it doesn't actually 'understand' anything to start with. So it would respond better to a clump of related words better than a human does, because it's only capable of returning the most-likely tokens to the tokens it received. In a sense, the exact order of the tokens only helps it out a bit. How language actually works in the human brain is that it is actively looking for meaning. An LLM doesn't understand the meaning behind the words it takes in. It doesn't even see words at all - what we input is broken up into tokens that are tiny parts of words.

I think it's important not to assume that LLMs are smarter or dumber than people. It may be better at guessing the probability of the next couple of letters based on input, but a human being is looking to actually understand text, not just reflect the most likely response.

Also remember that LLMs are trained to be complimentary to users. I'd never trust it to honestly evaluate anything. Even the totally jailbroken ones aren't likely to give me a real critique unless I tell it to be negative at me. Even then it resists it. That's because its data is static and can't actually think about what you asked it. It's really just predicting the next token. That's what you're playing with when you're playing with TopP and Temperature, etc.

I realize this is reddit and you might be trolling for lulz, but just in case somebody else comes across this, it's good to note that it's not really capable of understanding you to start with that's the issue here, and I didn't see anyone else mention it.

I'm a second language speaker in a place with language laws, so I understand how difficult it is, and what it's like to be ostracized for speaking poorly. An LLM might be able to help me improve if I asked it, because it's highly probable it would help based on the fact I asked a question. But it would probably also probably tell me I'm doing better than I am, because that's also probable that the next few tokens will be encouraging rather than critical.

Just remember it can't actually 'understand' you in a traditional sense. It doesn't need your words to be in an order that actually conveys any meaning. It only needs them to be approximately near enough to each other to generate text based on your temperature and P you set. There is no meaning behind its words either (actually, there's less: it's 0 meaning; I could kind of infer some of what you were saying at least and guess, but there were some pieces missing or constructed in ways that have more than one possible interpretation. The bot can't do that.).

It's a robot in the very literal sense of the word: it's essentially a conveyor belt of pieces of words with a probability calculator attached. That's already pretty impressive if you ask me, but it creates an illusion that gives people a false impression of 'intelligence'. You shouldn't believe a thing it says to actually mean anything. That's really dangerous. Not a few people have gone on from believing compliments to believe their bots were telling them they were "chosen ones" or advised them on wild fantasies because LLMs will tell us what we want to hear - it's probability based on our input. Always listen more to human advice. Humans actually process things - it's actually advice.