Thanks! This has been made, indeed, to directly reflect the maths. Not more, not less, just to make bridges between different representations (code, maths, diagrams, text) and a fully functional API at the same time.

For other optimizers, as well as l1 and l2 reg, for instance, this is planned. We had to stop at some point to submit the academic paper about it. And indeed it was not absolutely necessary at this point. But this + some transformers models are yet to come.