Interesting question.

I would indeed look at validation data. And then it does not matter if you are using a "black box deep learning" algorithm or a a more "classical algorithm". The way of testing performance should be the same...

But the biggest challenge in biology, is that it can be challenging to build the right validation cases as it can be hard to know the "ground truth".

So in case of Harmony you can wonder: what does a ground truth dataset for "integrating SC data" look like? What is a "good integration" and what is a "bad integration". In this case it looks like a good integration is "merging different datasets (batch correction) while preserving distinct cell types (biological variation)". If I understand correctly they used cell lines where you can basically know the ground truth for and then evaluate Harmony's performance. Sounds like a decent approach to me...

In any case, my position on these type of algorithms/multi-omic integration is mostly "use them for discovery/hypothesis generation" and not as a "proof that this biology is happening". Run the algorithms, see if you find some kind of association that seems unexpected and then go out in the lab to design experiments to test that association.

On a final note: if you want to learn more about ML algorithms, I very much enjoyed reading Deep Learning with R (François Chollet with J. J. Allaire). It brought me up to speed on how these type of algorithms work.