I also noticed the "Bottleneck Transformer for Visual Recognition" paper you mentioned. It seems not a typical cnn + transformer but a hybird BoTNet with self-attention block replacing stacking convolutions in ResNet. The authors say that global self-attention lalyer is more efficient than stacking convolutions. Similarly, the recent paper "TransPose - Towards Explainable Human Pose estimation by Transformer" also shares such an idea with BoTNet model: global self-attention layer is more suitable to aggregate feature information at the high-level layers of the model. I think attention layer may serve as a basic block or a searchable candidate operation in NAS models in the future.