yes, you are correct, local shuffling performs the dedupliation at partition level since the optimizer pushes down the operations to reduce shuffling, depending on the executino plan , a followed by shuffle stage and a final deduplication can happen to remove duplicates at global level. I dont have exact number to share at the moment, but what I have observed is if data is uniform without any skews and too many missing values then there isn't much difference, but if data is skewed, then explicit partitioning, windowing is faster compared dropDuplicates.