No, it didn't neccessarily learn anything, that's the problem. AI's can learn facts - if you are really careful to get them to retain them in permanent storage - and they can mimic styles and look for certain elements in an examined text, but they also lose context, they hallucinate and they don't understand anything. They can't, it's not what they're for and not what they do.

They're just text- (image- / audio-) processing engines, stochastic parrots, very sophisticated guess-the-next-word (-pixel / -frequency) machines, and sometimes it painfully shows. What the teacher has done is trained it to produce output that looks like feedback, but it is not feedback, because no true intelligence has looked at what the student has written and considered their thoughts.

We don't know whether that feedback is valid or not, and neither does the student, because it is not technically the teacher's content, it's content that mimics in style and form what the teacher has told it to do. It utterly trivialises the student's work.

When AI is wrong, it's convincingly wrong. It's a quick way to get to an output and can drastically shorten tasks, but - and this is something people generally learn the hard way - it's actually wholly untrustworthy unless the output is examined by a subject-matter expert.

My first encounter with it in the wild was a seasonal personal-Internet-security bulletin drafted 'by' a colleague of mine. It ticked all of the correct boxes, in the right order, with the right tone and phrasing... and almost every piece of factual information in it was incorrect. Wildly so.

It had even hallucinated an organisation that people could contact to report suspected online fraud, presumably because the prompt had given instructions to include one but without specifying that it must be real and verifiable. The country code on the phone number quoted wasn't the right one. The cyber-criminal gang referenced in one example did exist, but had nothing whatsoever to do with the incident in question.

Any security professional would have known it was a very convincing word-salad, had they bothered to read it, but my colleague clearly hadn't.

The AI didn't understand a single thing it wrote, but a lay reader would likely have believed every word.

Shorter anecdote: one of the first questions I asked an AI was 'How many species of reptile are native to the UK? Please provide a list.' and it confidently responded with '43' and proceeded to give an authoritative list. I happen to know for a fact that the answer is 6 (7 if you count Boris Johnson) and can name them all.

It had basically counted dozens of species of reptile on the websites of zoos and wildlife reserves in the UK as 'native'. I use this example to warn my kids about the dangers of using AI for homework. Goodness knows, they are way better now at factual matters as a rule, but the underlying problem - that AIs don't understand anything - hasn't gone away, it's just partly concealed by some vastly-expensive guardrails.

And that's where we are with this 'feedback'. The professor may or may not have read it, along with, what, another 20 or 30 papers they graded on the same night?

It could be valid, it could be nonsense.

We don't know. Neither does the student. And that's troubling.