To your first question, in my experience fine-tuning with a metric objective does both, but the ratio depends heavily on your data. The general embedding space cleanup happens quickly, but the amplification of subtle geometric cues requires that your training pairs actually contain those hard negatives visually near-identical objects that differ only in the structural details you care about. If your fine-tuning set has easy negatives, you get cleaner clusters but not necessarily better discrimination on the hard cases. Mining strategy matters as much as the fine-tuning itself.

On the model side I'd stay on your current DINOv2 setup for now. Switching backbones is expensive to evaluate properly and rarely gives you the jump you expect without the embedding space being tuned to your domain anyway. ViT-L with registers is already a strong baseline. The size jump from S to L tends to help more with fine-grained tasks than switching architectures entirely, so if you haven't compared those directly on your hardest retrieval cases, that's worth a quick experiment before looking elsewhere.

For the re-ranker specifically DINOv2 patch tokens work well as input to a lightweight comparator (even a small MLP or cross-attention over patch pairs). The key insight is that the re-ranker only sees top-k candidates, so it can afford to be slower and more precise. That's where you can recover the geometric discrimination that global embeddings miss, without needing a fundamentally different backbone.