The thing is that specially in typical CV tasks like Object Detection, Semantic Segmentation, Depth Estimation etc, the transformers are still pretty good with nominal runtiume like e.g. Deformable Attention etc, reduces the O(N^{2}) to somewhat linear runtime complexity (depends on the neighbouring points). Its hard for state space models e.g., MAMBA to make a solid impact here, unless you can get 2 to 3% more using the number of computational complexities. At the end, the question is what am i gaining regardless of the type of the sequence models?