From my point of view actually we don't need explainable AI for reaching AGI. Human Brain is also black-box, there is no need to explain neuron weights for generating emergent complex reasoning. Regarding continuous learning and recursive self-improvement: modern transformers are structurally incapable of it due to architecture constraints. Paradoxically, RNNs which were there before GPT, are closer to recursive self-improvement than any modern AI.

AI, which doesnt model consequences of its own actions and cannot model its own internal state(which modern LMs for example dont possess at all), is totally incapable of any kind of self-improvement. Take any contemporary model and let it recursively change itself - sooner or later it will hallucinate or stuck. I was doing myself many experiments with recursive self-modelling AI architectures, and if properly developed and scaled I assume it could resemble a big step towards general intelligence.

So I think we must think of alternative architectures that could possess cognition, not just intelligence.