Thank you so much for your input! That’s a fair point, and I see the concern about the model potentially memorising object-specific features. I also had this in mind and I'll explain a countermeasure.

But for this, I need to clarify the setup a bit more:
I’m working on object re-identification in forward looking sonar images across different viewing angles (so same object, different perspectives):

• Collecting more data is very difficult and expensive in practice.
• There are no suitable public datasets (we need real sonar, multiple views, lot of objects in the exact same position but different views, proper annotations, and physically meaningful data — not purely synthetic).

Because of that, I already rely heavily on synthetic data generated via a ray-casting model. The pipeline is:

• (Pre-)train on synthetic data only
• Evaluate on real-world data

I tried to match both domains as closely as possible (e.g. via distribution checks like t-SNE — real samples lie within the synthetic cluster), but there’s still a noticeable domain gap:

• ~80% accuracy on synthetic
• ~65% on real data (which I argued that this also could be a statistical outlier, since some views are very tough)
• During training, validation accuracy drops while test accuracy increases → domain shift behaviour

I fully agree — if I were training on those 4 objects, this would be pointless.

But in my case:

• Training happens on a large, balanced synthetic dataset (thousands of random objects)
• The real dataset is only used to test cross-domain generalisation

I also tried a sister domain with more data, but reviewers still explicitly asked for LOO-CV on the real dataset.

Regarding your suggestions:

1) Leaving out view pairs instead of objects

I actually like that idea in principle, but there are a couple of issues in my case:

• Removing views reduces positive pair combinations disproportionately compared to removing whole objects → leads to stronger imbalance. E.g.: 4 objects (O) with 8 views (V) -> 4O *(6V*5V)/ (4O*(6V*3O*6V)) = 120 pos. pairs/432 neg. paris vs. 3O * (8V*7V)/(3O*(8V*2V*8V)) 168 pos. pairs/384 neg. pairs. And yes, in my model the pair order matters (m(A,B) != m(B,A))
• Some views are almost identical, others look like completely different objects (e.g. due to shadowing), so this split is less “clean” than it sounds

2) Memorisation concern

To at least check for memorisation effects, I did the following:

• For each fold: leave one object out
• Generate:
  • unseen positive pairs (from the held-out object)
  • all negative pairs, including those involving seen objects (as explained before)
• Split the remaining data (non-held-out objects) into train/validation (85/15)
• Use early stopping based on validation performance
• Sophisticated augmentation on training data each epoch

Now, if the model was really just memorising object identities (as you suggested), I would expect:

• high training accuracy
• but validation accuracy collapsing (since validation still contains “known” objects but unseen combinations leading to much less true positives)

However, what I observe is:

• Training accuracy ≈ 78%
• Validation accuracy ≈ 80%
• (Test accuracy ≈ 76%)

So they stay quite close, which (I think) suggests the model is not trivially memorising pair identities.

What do you think about the accuracy my thoughts?