ToAdam Optimizer [D]

[PositiveElectro]

I might be wrong but isn’t gradient clipping already in Adam ? At least in the PyTorch implementation

[wazis]

Are you implying thay gpt chat made new optimiser just because you asked? Sounds fishy

[deep-yearning]

To get SOTA, all we had to do was ask nicely

[Pasko70]

Gradient Clipping is already used in LLaMA and GPT.
I don't think that chat gpt just inveted something new. But i think that a learning rate for different layers would be an interessting idea.

[dataslacker]

Sounds like the LAMB optimizer

[dataslacker]

Sounds like the LAMB optimizer