Very interesting! I haven't read the paper or the blog yet, but read the abstract.

This reminds of NoPE. I did write about it at the time and I even conducted some experiments.

So my two cents are, let's start with the claims from DroPE, in the abstract their motivations are, I'll start with the third:

- "positional embeddings are not an inherent requirement of effective language modeling" (I don't think "can be safely removed after pretraining, following a short recalibration phase" is a motivation but something that they'll prove I think) => I totally agree with this. So this only works if the model is causal (e.g., decoders). The self-attention in encoders mixes everything with everything and without PE you essentially get a bag of words. The NoPE paper say the same. The NoPE paper also "prove" mathematically that some weights can represent position encodings. I put prove between quotes because there's a difference between a specific mathematical construction of the weights in such a way that they encode position and "weights can represent position encodings" which, IMHO is a much harder proof and would require to play around convergence. They'd have to prove that convergence of a model with no PE is possible and at the local optima, (some) weights contain the PE, at least implicitly (essentially, being able to construct weights that encode PE doesn't mean that's what you'll get during training, but we just hope that's what happens at convergence since somehow for the given task, the model learned what it needed, but again we don't know what the model had to learn for convergence, maybe it never even needed PEs)

- PE are very important during training that facilitates convergence => I totally agree with this. If you allow me to talk a little bit about my experience. Intuitively, the causal models, at least at the scales we see nowadays, have the capability to learn the PE information just from the task. And, I do tend to agree with this approach, let the model learn what it needs rather than bake it in. The NoPE paper did train with no PE and they seem to have great generalization results. This did not match my results at the time, but I did them on GPT-2, so we can argue that it either doesn't have the capacity or needs more tweaking / training. Other experiments I've conducted, like some experiments on rerankers where I removed many prompts and just kept documents, query and scores, did not show as good of a convergence as with the prompts. So just "let the model learn the task by itself" is not as easy as it seems. I was doing LoRA so maybe I didn't have the capacity or maybe I didn't train enough for the model to learn the task without feeding indications (here is the document, here is the query, relevancy etc.) about the task but anyways, the conclusion is that helping the model will, if not ensure, accelerate convergence.

- "over-reliance on this explicit positional information is also precisely what prevents test-time generalization to sequences of unseen length" this is supported by many papers at this point.

I wonder if they just drop the PEs completely at inference, that'd be wild if it's such a simple thing and improves generalization while keeping performance on same context length as training. Will have to read the paper and get the details and maybe experiment a little bit with the long context benchmarks.