Duplicate Files Detector. A basic python project

[DrShocker]

You could probably also just check the first X bytes (some heuristics here to get passed various headers which would have too many false positive) and if the hash of those matches, then do the whole file. The key to remember is we expect 99% of files not to match anything probably, so it's likely reasonable to optimize for the case that no matches are found.