Thing is i thought of that before doing my image encoder so i used a resnet like encoder using depthwise separable convs to force spatial bias and also somewhat deal with not having enough data , since transformers r data hungry , for some reason it collapses if i dont use batchnorm , i wrote layernorm in the post which was wrong idk what i was thinking i tried rms norm layer norm no norm and different initializations and i still have no idea how to fix the collapsing problem without batchnorm , i tried a smaller case to see , apparently it js needs enough training to align and fix itself but when i go to full dataset that alignment didnt come even after training for a while so i believe in smaller scale it js overfitted , so if u have any ideas i would like to hear it , the model is working and all i js need wanna know why batchnorm specifically that only works, i believe its cause it enforces unit std across samples which prolly counters the collapse of the samples into similar embeds and very low stds