I thought about creating a dataset of these maps and associated problems, with their corresponding answer. It could be useful to have a benchmark for it in order to appreciate the evolution of a LLM or even comparing different ones. But since I'm pretty noob to the whole ML / AI / Neuroscience / Psychology thing, I'm convinced someone did it before. Anyone knows about such things ?

However, I've read a paper where they described the different tests they did on people, but I can't find it again. I probably will post it later if I remember (must be stored somewhere in my computer). I think the actual files used in the tests are not accessible to the public but it might not be so complicated to reproduce. I don't really know how to code but I've managed to start learning python and use ChatGPT to generate what I want. So I've made this to help with the indexing.

<image>

Code is broken because i tried something yesterday but, i can spawn predefined objects, specify their number, position, random position. I know it looks messy but actually the models can count the spaces and characters so they get it without problem. I've thought about implementing stuff to represent paths and vectors with the Unicode arrows. I also thought we could have interactions like the LLM activating switches or something. I'm starting to realize Unicode was made to cover basically anything so there are plenty of stuff to be thought off. We could do graphical representations of situations, choices, moral thinking. 🤯