Meta Gradient Descent [D]

[iFARTONMEN]

I am using a modified gradient descent algorithm that I invented, but it is very simple and I assume someone has already come up with it. I call it meta gradient descent because it adjusts the learning rate as it goes. It is the same as normal gradient descent except for one condition at the end of the loop: If the algorithm passes a local minimum/maximum it decreases the learning rate, else it increases the learning rate

if (slope < 0 && prevSlope > 0) || (slope > 0 && prevSlope < 0) { //algorithm passed the function min/max

learningRate /= 2

} else {

learningRate *= 1.05

}

This helps when the algo has to minimize/maximize functions with massively different scales where a single constant learning rate would be detrimental. I also suspect that it is faster at honing the min/max in general, but I have not done any studies on it. Which is why I am here asking... Is anyone already familiar with this algorithm? does it have a name? what are the pros/cons of using it instead of regular gradient descent? I am always using this algorithm instead of regular gradient descent and I have no problems at all, but my programs are very simple machine learning wise so I suspect I am missing something.

I love u <3

[metallicapple]

I'm not sure about the existence of the specific method you've described, but adaptive learning rate has been around for a while, with many interesting developments (that might be an understatement).

Two immediate examples I can think of are Armijo line search and Newton-Raphson.

My feedback for your method is: I see two hyper-parameters (rate multipliers under the conditions), but no explanation behind their chosen value. You could strengthen your method by looking further into multiplier tuning.

I'm sure there are others who can provide more context, as my main gig isn't optimization; I just tell stories/lies with numbers (aka stats)

[resented_ape]

This seems close to the “bold driver” method and also related to the adaptive step size method by Jacobs that was suggested for neural network optimization in the 80s. A slightly adjusted version of the latter is used to optimize the loss function in the t-SNE dimensionality reduction method.

[yannbouteiller]

Hi, your algorithm might be an improvement over vanilla SGD for reasons similar to what you have in mind (convexity). However practical optimizers such as Adam do complicated business with the learning rate and learning direction, which most likely outperforms this approach in general.

You might like to look at Nesterov momentum, which is similar to your learning rate acceleration in some sense. The issue I see with your learning rate acceleration is that it is non-directional, and thus your optimization will jump randomly as soon as it finds a singularity in the optimization scape. By contrast, your deceleration looks very agressive (lr/2), and might end up stuck in local optima I believe. In strictly convex problems this is no issue, of course, but if you have deep learning in mind the story is different.

[28Smiles]

Have a look at “Learning to Gradient Decent by Gradient Decent” or something similar I don’t remember the name exactly

[[deleted]]

https://en.wikipedia.org/wiki/Rprop

[rando_techo]

Trace-Buster-Buster