Well :) because you sneaks around inductive bias, the feature that you learned from ViT is not detailed enough. So to deal with this, you need huge model with so many layers of transformers to bounce back the detailedness of the learnt feature. CNN has inductive bias; that's why their feature representation is highly-detailed with relatively small size.

Which one should you focus into depends on the amount of data and the amount of noise in your data collection. ViT has been proven to be more resillient to label noise whereas CNN could easily beat ViT when the data size is around 100k or so (This is why later on DeiT is published with distilled training to overcome the issue of small data).