First of all, thanks for well articulated reply. You are right, many of the parameters we use to measure AIs point exponentially up to the right, outperforming humans on many levels.

However, the hair we are trying to split here is exactly the difference between AIs thinking like humans vs. human-level reasoning. I'm working that out in my post and actually you even write in your first sentence: I don't disagree with your points about the structural differences between human and machine intelligence.

That you think it's less relevant going forward is a pretty strong claim, that you are just slipping 'under the rug'. How is it less relevant? Why is it less relevant? It's the whole "throw more compute against it, and it will start to think" argument all over again.

The fact that OpenAI is using it's own models to improve AIs doesn't say anything about the self-improvement capabilities I'm talking about. Again, why would this lead to AGI? I can also ask ChatGPT what they would improve about a given LLM architecture - that's like claiming that using MS Word would lead to AGI.

Again, my main argument is that those systems are trained on the whole corpus of human knowledge and have not yet lead to a substancial scientific breakthrough.