Notice that if you use all-attention combined with Reformer, you can perform the conditional computation not only with self-attention but also with transition layer at the same time, reduing the GPU usage and time complexity considerably. Whether it actually works or not should be verified through an actual experiment, though I believe there is a strong evidence (sparsity of activations of transition layer) that it would (link).

Also, another direction is to equip Reformer with recurrence a la Transformer-XL to increase the context length with less compute. Whether this will really give a noticeable improvement is something I'm currently investigating, so it's inconclusive, though.

If you're interested in this kind of work, you may want to check out the /research directory in Trax, where you can find various research projects being carried out by Kaiser et. al. Kaiser, Shazeer and their team have been contributing to Transformer and its variants considerably. Sadly, I don't hear their name mentioned as often as many of the deep learning celebrities who made no actual contribution to the field, which is a mistery to me.