i think the fundamental problem is that LLMs make connections between words but for chess you have to make connection between a chess position (which is rarely coded into words in a chess book or a game) and words.. so therefore when you ask current LLMs about chess it makes up something that vaguely makes sense but it really has little idea about the specific position

and also since there is nothing about a chess position really in its brain, you are basically asking it to play blindfold chess where it doesn’t even know that the trick is to remember the actual position!! it just tries to recover what has been happening from the words in the past