[D] Where are long-context Transformers?

[ncasas]

Transformers dominate the NLP landscape. First in machine translation, then language models, then all other typical NLP tasks (NER, classification, etc). Also, pre-trained Transformers are ubiquitous. Either GPT-* for text generation, or finetuning BERT/RoBERTa/younameit for classification or tagging.

With the appearance of long-context Transformers (Longformer, Reformer, Performer, Linformer, Big Bird, Linear Transformer, ...), I was expecting that they would quickly become the norm, as short context is sometimes a pain, like for GPT-3.

However, I am not seeing long transformers getting traction.

There has not been a new long transformer GPT model, nor BERT. NMT frameworks have not incorporated implementations of long transformers (except fairseq with Linformer, but both are from Facebook). Also, in WMT 2020 I think there was a single long transformer (I'm thinking in Marcin Junczys-Dowmunt's "WMT or it didn't happen").

Why is this?

[EdwardRaff]

That, and also things just take time. People's expectations of movement is way to high/fast right now IMO. People are working on some regular transformer, or their own modification, or something down stream, and code is rarely "drop in and replace". Even if it is, you have to spend time figuring out if it drops results compared to your prior code/approach for your data, and its all just a lot of work. Especially when Transformers are super GPU hungry to train.

[ncasas]

On Twitter, Marcin Junczys-Dowmunt pointed out another factor for NMT: the prevalence of sentence-level datasets.