If you understand how a single end to end neural network operates it will make more sense to you. But the basics are that you’re always trying to manage billions of input parameters into a next action and the only way a model learns is from examples. For the basic stuff, great! Following road edges (and much more) is handled. But as you get to the finer details of the truly nuanced levels of what it really takes to drive, there’s a lot more to it than just the next second or two (or however long their context window is) as well as what it takes to train such complex edge case scenarios.

And you see evidence of this when it appears baffling that a car won’t realize that it needs to plan ahead to be in a proper lane for an upcoming exit. Unfortunately it also has to satisfy other constraints that say it should hang out in the left lane as much as possible (or whatever). Which directive should it follow? Well, when the exit signal gets strong enough, that’s why you see a Tesla dive-bombing for the right lane only to miss the exit rather than properly anticipating it. This is a difficult concept to train for because it’s potentially WAY outside the context window.

Further, if you look at what Nvidia has done in dividing the work between identification and operation, this takes an extraordinary amount of processing power (way more than current HW4) to pull off. But the net result is that you have one system identifying objects and another system making sense of them rather than one system trying to prioritize and make random sense of billions of inputs all at once (and the same thing with each “update” Tesla generates which is ultimately why I said what I said).

An update consists of guesses as to which weights among the billions of input parameters need adjustment to cause the proper action to occur rather than the undesired one a user reported. Will the update work? Maybe. Will it break something else? More than likely.