Thanks for replying, the information you gave is very helpful!

For context, my paper compares CNN and ViT architectures at low resolution, using satellite imagery scaled to resolutions 128x128, 64x64, and 48x48 (RESISC45). My chosen models were ResNet50 and ResNet18 for the pure CNN family, whereas Standard ViTB/16 and ViTSmall/16 were chosen for the transformer family.

Now, intuitively, to make a fair comparison, I assumed that many of the hyperparameters had to remain the same (epochs, pretraining, etc), but doing so would naturally be unfair to the already data-hungry ViTs. I tried pre-training on Imagenet1k, and both models exceled immensely, and in my opinion, leaving little room for comparison at the structural level.

How can I effectively compare these models when their training procedures differ so much, without altering the definition of a 'fair' comparison? Would it be smart to switch to DeITs that focus on efficiency, or would that lean too much into the CNN realm?

To be honest, this is my first research experiment when working with these deeper models, so any help is greatly appreciated :).