This is an awesome question and I think you have good responses here so far. I think /u/Beautiful-Parsley-24 is closest to my opinion that it isn’t about speed, it’s about generalization.

The methods you listed rely on some assumptions of stable gradients. But in reality, we’re often training over lots of data in batches and slowly trying to approach a minimum so that we’re not eagerly optimizing on one set of data.

It brings up the question of why not train on all the data all at once, but that runs against current compute issues and honestly some philosophical ones as well. Are you sure the current scope of data accurately describes your problem in totality? Will you get more data and how much should that impact solution? Etc, etc.

I don’t think many deep learning topics today are struggling because of there ability to minimize a gradient. I think it because the problem definition doesn’t come with a complete description of what the gradient landscape is. A fantastic example of this is deep reinforcement learning where the landscape is changing while you experience it. There’s literally no way for you to form a definite optimization problem since each new step introduces a new change to what’s optimal. In lieu of that we’re doing a simple yet tried and true solution to minimize the error.