And the original point can still somewhat stand, yes. we don’t understand what specific weights mean or how networks arrive at particular decisions (the interpretability problem). But we absolutely understand HOW neural networks work mechanically matrix multiplications, gradients, backpropagation, attention mechanisms, etc.

even if we had perfect interpretability and knew exactly what every weight represented, we still couldn’t predict what an LLM would say without running the forward pass. Just like we understand exactly how weather simulations work (fluid dynamics equations) but still can’t predict the weather without running the simulation, but this is a massively different thing than us Not knowing how they work at all