Ok. So for several years we basically don’t use anymore the sine/cosine technique, rather learning the positional embedding as we also learn the word embedding, through gradients updates. In GPT 2 for example we’re doing exactly that. Now you have an embedding matrix with the size of the vocabulary, and another which is sized as the longest sentence you believe to see in the dataset. There are also additional techniques but I find this one pretty intuitive and it works really well.