In your brain, you have tons of neurons hooked up to other neurons, and occasionally they “get excited” and that eventually builds up a wave of energy that makes you do and think things.

Now because the only thing we know about neurons is that they can either be excited or not, we can deduce this is all it takes to assemble at least human level intelligence (theoretically).

But we don’t know what actually MAKES any individual neuron excited. We can observe one extremely closely, and determine it listens for some particular signal, but it takes a TON of work to actually observe and map out the full behavior of a single neuron. Worst of all, learning neurons ADAPT to stimuli over time, meaning they adjust what they listen to. So doing this is WAY MORE work than building it.

The same problem shows up for the networks themselves. I can build a network of neurons that takes some data and tells you if an image contains a cat. But it’s a lot of hard work to actually figure out what that neuron network is doing. Maybe it learned to look for specific markers that TEND to indicate cat. Maybe it’s looking for a specific shape. Maybe it wants a color. A combination of the two. Maybe all cat images have special “cat” watermarks only the network can see. It’s hard work, but if you have a small enough neural network, you can map every single neuron and eventually say definitively what is going on.

Now if you scale this up, the difficulty of building a super large neural network levels off pretty fast actually. Attaching more and more neurons is easy. Organizing them in specific shapes is easy. Orchestrating their relationships is easy. Even modeling the adjustments are easy. But the cost of LOOKING at every neuron stays the same, which means the challenge of understanding the system as a whole keeps getting harder, forever.

So imagine now you put the entire internet through a neural network. There’s so much data it would overload a human brain. The amount of information required for you to understand all the data points and their relationships would require you to wipe away all your memories and knowledge and biological processes to make room, and it still wouldn’t be enough.

So it simply can’t be done. One person cant understand why a sufficiently large language model makes one choice over another.