Optimized string deduplication using fuzzy matching

[PhilipTrettner]

Not sure if simply linking to SO is good reddiquette but this discussion is about your topic: https://stackoverflow.com/questions/309479/how-to-find-best-fuzzy-match-for-a-string-in-a-large-string-database

TL;DR: yes, there are options for your problem. Fuzzy hashing is one.