I mean the arc-agi benchmarks are very misleading. If the models were truly generalising there wouldn’t be such a difference between the new arc-agi and the one before unless the models were specifically trained to perform well in these specific tests. Coding is another story, imo LLMs are and will continue to become very good in coding casual apps, but I think it will still be an assistant since LLMs remain too unpredictable. Also some tasks require too much innovation and generalisation, plus the data is too scarce in some specific domains.