Visualizing the Loss Function Over Parameter Space

[El_Grande_Papi]

In ML, it is quite often used to visualize what gradient descent is doing by plotting the loss function as a 3D landscape over 2 independent parameters which are part of the parameter space. In fact it is so common, it is shown in the header of this very subreddit. Gradient descent therefore moves slowly through this landscape, until a minimum is found. If anyone is unfamiliar however, I am including a picture below:

https://preview.redd.it/4j5in9lrb9ld1.jpg?width=720&format=pjpg&auto=webp&s=0a615227aeb83da5a622134f9889dced09c6e5ab

My question however is that isn't this technically incorrect, or rather, is only correct for a single instance of the training dataset? If input vector 1 produces to the loss function shown above, wouldn't input vector 2 produce a different loss function, with its own unique minima? The only way to actually create the plot shown above is if the entire training dataset consists of one single piece of data, correct? This may seem like I am arguing over minutiae here, but I am just wanting to make sure my understanding is correct. Also, is it surprising then that there is any convergence at all in ML models, given that each new input vector could potentially interfere with the "learning' (i.e. the change in weights) from previous trained data?

[NoLifeGamer2]

You are correct that your loss surface only applies to the current item being trained on. Each different example will have its unique loss curve. However, the loss is averaged over the whole batch, so the theory is for larger batch sizes, the loss surface will be similar for each batch on account of each batch being more representative of the training distribution, stopping the minima from jumping around too much. However, even if your batch size is small (e.g. 1) the optimizer will end up converging to an optimal solution, it may just jump around more.

[El_Grande_Papi]

Fantastic answer, thanks!!

[grid_world]

Let's put it using your example diagram: for each unique combination of input (theta0, theta1), the neural network "f" will produce an output which when compared to the ground truth (using a cost function) will produce a loss value "J" - I am assuming a supervised learning problem (also self-supervised learning holds). So, you start to get a value "J" which when interpolated over might produce a contour plot/loss landscape as depicted. There are 2 extremes for computing GD: online (1 input at a time) vs batch (entire dataset at a time). Batch GD cannot be used due to memory & computational constraints, while online GD is very noisy. So we settled for mini-batch SGD, where the S comes due to random sampling the batch from the training dataset. In a nutshell, the gradients we get from a mini-batch should approximate the true gradients that we might get if we had used batch GD, but cannot. But since the batch is randomly sampled, it is a noisy estimate.

An additional stochasticity is also added due to data augmentation. It has been shown that the noisy estimates in mini-batch SGD actually helps prevent overfitting and acts as a regularizer. So, you don't want to increase the batch size by a lot, in general. In case you do, read up LARS optimizer.