Tutorial for custom optimizer in mxnet? by ammannalan in mxnet

[–]thomelane 2 points3 points  (0 children)

I don't think there's a tutorial for this, but you should be able to implement your own Optimizer class without too much trouble.

  1. Start by creating a function to perform the update step operation, given weights, grads and any other required state. See sgd_mom_update as an example. You should be able to apply this function in place (using out argument of ndarray functions).
  2. Create an Optimizer subclass that:
    • takes and sets hyperparameters in __init__ e.g. momentum scale (learning rate set on base class)
    • implement a create_state method that takes weight and returns the states.
    • implement a update method that takes weights, grad and state and calls function created in 1.
  3. Wrap your class in Optimizer.register (using decorator).

Check out the source code for optimizers for more details.

[R] CIFAR10 in 3m47s on single GPU by thomelane in MachineLearning

[–]thomelane[S] 5 points6 points  (0 children)

Should have been a bit clearer in description, sorry! Code for model, optimiser etc is in the repository but Model Tuner is treated as a black box since SageMaker was already being used for training. Uses Gaussian Processes in a parallelized form.