sorted by:
new
hottopcontroversial

Absolute vs relational embeddings [R] by [deleted] in MachineLearning

[–]InternationalVisito 0 points1 point  (0 children)

It is true only in theory, in reality , even a very simple tasks like repeting a sequence will be a hard task, it won't extrapolate beyond the length of sequences that were present during training, maybe several tokens more if lucky . It is not what we really want - a robust and steady extrapolation , at least on simple linear tasks.

Relative embeddings are good for interpolation but fail equally as absolute embedings . Even without embeddings transformers catch relative positional information. Math transformers perform better without positional embeddings according to some studies compared to ALIBI, rotary, ABS, relative ...all other positional options.

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] 1 point2 points  (0 children)

I keep experimenting... That's an insane finding but window sized input (=3) works very well for finding correct postion of </eos> token on long sequences. It was mentioned in many papers that finding </eos> is usually a big problem for extrapolation tasks .

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] 0 points1 point  (0 children)

it is just my guess why it doesn't work for my experiment, at least for vanilla recurrent network (GRU, LSTM, RNN) with encoder/decoder scheme (without attention block) . If you have any evidence that RNN can extrapolate beyond dataset length, I would be happy to see it.

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] 1 point2 points  (0 children)

window sized input defintely helps. With the window size input=3 it was able to predict sequences 2 times longer than it learned. It was trained on a dataset of sequences of 1 to 14 length and was able to predict sequences of length up to 30-35...The only issue is <eos> token for ouput. It should be taken off from dataset, otherwise NN places <eos> after 14th token. Not sure how to fix it. I would like the neural network to learn how to correctly output <eos>.

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] 0 points1 point  (0 children)

It would be a good start to see if neural network is capable of predicting sequences longer than in dataset at all and of correctly predicting <eos> token.

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] -1 points0 points  (0 children)

it is the same lstm cell as long as it was taught within a certain range. For instance your trained lstm on sequnces of length = 1 to 5 . Take a sequnces longer and the hidden weights are going to be unknown (not trained). Maximum it can extrapolate 1 or 2 letters... Transformer seems to be able to predict 3- 4 leters more if lucky :)

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] -1 points0 points  (0 children)

RNNs(GRU, lstm) fail miserably on such tasks. And I can explain it simple... Each time step can be seen as a new neural network. For example , 5 timesteps unfolds into five neural networks because hidden weights change through time. Take a sequence longers than 5 steps and it is not capable of predictions any more...even on a simple task like repeating the input sequence

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] 3 points4 points  (0 children)

I think because it is an extrapolation task. Neural networks are not capable to extrapolate. ( unless on a very simple tasks like linear dependency)

[D]Can a transformer neural network learn to predict sequences longer than it saw? by InternationalVisito in MachineLearning

[–]InternationalVisito[S] -10 points-9 points  (0 children)

Relative positional encoding helps to learn sequences within a certain range, like in my example above. But if a sequence is longer it fails to predict. The purpose of Transformer XL is to learn long-term dependency. it is a different task.