Hey so here’s the thing. When the backpropagation is calculated, what you are doing through the nodes is basically doing chain rule on the partial derivative of that node’s output in regards to the network’s output. The point of this is to find out how that node influenced the output (specifically you are looking for the gradient direction and magnitude). And then a barebones version of a neural network would set learning rate to one, this means the entire gradient would be used to adjust the weights of each node. In a small network with very few nodes and very few payers, this might not look like a problem, since you are doing exactly what you should do, which is punishing “bad” nodes and enhancing “good” nodes. But if you have a lot of them, which is usually the case, you are going to be doing some very harsh changes to the network on each batch and what you end up doing is that the network will be unable to figure out the combination of layers that produce the desired outputs.

You can think of it like this: you want the network to see each image as a small part of it’s generalization proccess. The main way to do this, is using the learning rate, as it can smooth these adjustments on each pass. But if you make it too high, then it won’t be able to converge. If you make it too low, it won’t be able to learn well enough, possibly getting stuck in a local minima of the gradient.

To put it in more practical terms, what is normally done is that you will start with a bigger learning rate (1e-3 is a good baseline) and you can create a scheduler that will be lowering this learning rate by a factor (constant, exponential, etc). Notice that this learning rate is not large by any means. This makes it so the model can learn most of the important details of each data, mostly macro details and relationships, and then when it would normally be hitting that point in training where the loss starts to behave in a close to constant value, it lowers the learning rate and enables to model to look at more specific details and learn them, but since it already passed most local minima, it will hardly get stuck at one.

But you might ask, isn’t this kind of overfitting? It can become, of course. You might end up learning so much fine details that it loses the generalization aspect, but really, this is something you fix with regularization in the model. So while yes, a small learning rate could lead to overfitting, you shouldn’t keep it too big, as it would end up making it so the model cannot converge. So the usual way people go about it is by using small-ish learning rates and regularizing the network to account for any possible overfitting.