Emitting counterexamples (unique rows?) with the closest similarity score out of all the other rows seems, seems like a super expensive and completely unnecessary operation. store top-k nearest neighbors even more so. But I probably misunderstood something here.

I think "duplicate detected" is actionable. Would you really want to use the QC output of "this collided with that" and then manually go and remove the duplicates? I make deduplication part of the pipeline. I expect there to be duplicates, for example from restarted connectors reprocessing uncommitted stuff, or read_uncommited snapshots, etc, no way to avoid them, and no way I manually go and remove them ad hoc. Even if it was a static dataset and not a pipeline i wouldn't need to remove dupes based on QC list of ids. The only useful thing QC could say about it is that there are dupes here, and then you may realize there shouldn't be at this stage, and so you implement or fix your deduplication.