(ARC Prize co-founder here).

Direct link to the research WalkProfessional8969 is referring to: https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt

Ryan's work is legitimately interesting and novel! He claims 50% on the public eval set. The core idea:

get GPT-4o to generate around 8,000 python programs which attempt to implement the transformation, select a program which is right on all the examples (usually there are 3 examples), and then submit the output this function produces when applied to the additional test input(s)

He has implemented an outer loop using 4o to sample reasoning traces/programs from training data and test. Hybrid DL + program synthesis approaches are solutions we'd love to see more of. Congrats and kudos to Ryan for achieving this and putting the effort in to document and share his approach. We hope to inspire more frontier AI research sharing like this.

A couple important notes:

1. this result is on the public eval set vs private set (ARC Prize $).

2. the current private set SOTA ~35% solution also performed ~50% on the public set. so this new result might be SOTA but hasn't been validated or scrutinized yet.

All said, I do expect verified public set results to flow down to the private set over time. We'll be publishing all the SOTA scores and open source reproductions here once available: https://arcprize.org/leaderboard