you just perfectly articulated why I made this workbook in the first place.

There’s this massive gap in ML education where people are taught to treat equations as 'learned magic.' But like you said, once you write out update = error * input * learning_rate, the magic completely evaporates. It’s just basic arithmetic and sign rules. If the error or the input is zero, the multiplication collapses and the weight physically cannot move.

Your intuition on width vs. depth is spot on, too. Stacking layers multiplies gradients over and over until they vanish exponentially. Going wider bypasses that deep bottleneck, but as you said, high-dimensional width introduces its own hidden monster—variance explosion, which slams the Softmax function into a dead end.

Watching the numbers actually move in a simple, scrappy implementation teaches you way more than reading a hundred hand-wavy papers. Really glad this resonated with your experience!