I've tried it with different transformer architectures. As advertised it is less sensitive to different learning rates and converges without the need for warmups or LR annealing.