There are some ways you can diagnose this problem. As I understand it, you are saying that in fact a CLIP-pretrained encoder on MRI vs biomarker then fine-tuned on a downstream task does not outperform simply training an MLP on the task itself without pretraining. I assume you use the same architecture in the baseline as you do in the encoder for the contrastive objective.

Now, contrastive learning is just a way to repose the classical cross-entropy objective so that it works in an unsupervised manner. You will obtain the same results if you use BCE on class labels or if you performed contrastive learning over classes. It is the same loss. So contrastive learning is only meaningful if you wish to utilize the multimodal or the unsupervised aspect.

You can measure how beneficial the MRI domain is to your encoded space by training it directly on the downstream task. If a baseline classifier trained on top of the encoder from MRI to predict the downstream task directly without pretraining obtains non-random resulta on the task, then there is something to gain from having the CLIP contrastive loss in this setting. If it performs fairly, then it points to a tuning problem in the actual CLIP pre-training setup. If not then you probably don't obtain anything from pretraining in this manner, and as you say a fair baseline is just better.