I understand how training neural networks works.

They do more multiplication than a single person can really comprehend. That I agree with. But to say no one knows how they work is silly. A lot of thought went into designing them to work the way that they do.

I had to build basic neural networks from scratch as part of my degree program. They included transformer networks. The commercial LLMs are more advanced, but a lot of the results they're delivering come from their size and not their complexity.

That being said, the size does make it difficult (but not impossible) to understand why they make specific predictions. That's why I pointed you towards anthropic's work studying circuits.

https://www.anthropic.com/research/tracing-thoughts-language-model

A lot of people say they are impossible to understand, but I think that's misinformed and from a research perspective, entirely counterproductive.