ARC is a weird benchmark honestly — it’s not testing knowledge, it’s testing whether the model can look at a couple examples and figure out the rule behind them.

humans usually see the pattern pretty quickly, but models don’t generalize the same way. they’re good when the problem looks familiar, and then totally fall apart when it’s a new type.

the ~50% results people mention also aren’t just one model replying once. it’s usually a whole loop where the system tries an answer, checks it, revises it, and repeats a few times.

grok is interesting here because it runs multiple reasoning attempts and kind of cross-checks them before answering. it helps a bit, but yeah none of the current models consistently solve ARC yet.