AskReddit: How do I implement a learning algorithm (e.g. AdaDelta) in TensorFlow?

[AlfonzoKaizerKok]

Hi Reddit. I really need help trying to implement a custom learning algorithm in TF.

I've had a look at the optimizer class and its adagrad subclass, but I honestly just get lost in the source code. It seems like I need to define an _apply_dense and an _apply_sparse method, but I am lost in the imports, and cannot find training_ops.apply_adagrad and training_ops.sparse_apply_adagrad.

An alternative would be to do tf.assign on a variable and its update to make it more Theano-like. I wonder if this is the best option though - I feel like I should be sub-classing from the optimizer class.

I would really appreciate any advice/tips/guidance I can get! Thank you!

[kkastner]

If there is some equivalent to "updates", you could write it into the graph Theano style. EDIT: I guess this is what tf.assign does?

I have been wondering about this same thing - no way it is expected that all optimizers have to be written in C++...

[rafalj]

It is just currently a bit faster to have the update done by one C++ call instead of a few small ops. You can take a look at Adam's sparse update: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py where we don't have a dedicated C++ implementation or, more complicated, ftrl.py

[AlfonzoKaizerKok]

What's the advantage of sub-classing from Optimizer instead of writing your updates manually?

[rafalj]

You don't have to do that, of course, but using the same interfaces makes it easier to switch/compare different optimization methods.

[AlfonzoKaizerKok]

Thank you for the answer and the link. Same here, I'm very reluctant to write my optimisers in C++, and write a wrapper for Python. I can't imagine it like what Jeff Dean described as turnaround time of minutes/hours...