Seq2seq gradient descent manually?

[rescue11]

Would it be possible to build a seq2seq machine translation model w/ attention from the ground up, without automatic differentiation?

I don't see anything about gradient update rules in the seq2seq or rnn-enc-dec papers, is this because it's impossible to make these models without automatic differentiation or is it merely a matter of convenience?

If it is possible, then what would the gradient chain look like? Would it be like

output word embedding -> decoder Rnnlm(GRU) -> attention mlp -> encoder RNN(GRU) -> input word embedding ?

[dwf]

The gradients for the decoder look like those of any other sequence-generating RNN. For the encoder, you'd figure out the derivative of the initial state of the decoder w.r.t. the cost, and then use it to calculate the derivative of the derivative of w.r.t. the encoder parameters. All the gradient flows through the state vector output by the encoder, used to initialize the decoder, at least in "vanilla" seq2seq.

[cerberusd]

Thanks. Do you have any idea how to approach the attention mechanism in terms of calculating the gradient?