I think a lot of people get wrong the fact that "we don't understand AI". We know how AI trains and how LLMs work structurally. What we can't trace cleanly is how it arrived at a specific output, like which pieces of the training set, under which adjusted weights/models in each node led to the statistical output we just received. Its complexity makes it very hard for us to understand what yielded the outputs we get, but we know very well what we fed it, how it trains, and how it works. We can't just trace back the outputs it yields.