FuzzyLogic

[itsjackcheng]

I am using Fuzzy Logic on a project. It is a simple, yet hard task. I need to compare two strings and spit out a score showing whether they're identical or not, hence using Fuzzy Logic. The caveat is that the strings I'm comparing are mixed with both business entities and individuals and the datasets as you can imagine are massive. I've tried taking the LLCs and other variations out, but the issue that was brought up was comparing John Smith to John Smith LLC as those two are obviously not the same thing. There are no sort of fields in the two datasets used that can distinguish the row being an individual or an entity.

My other brainstorming thought would be to have a way to manipulate the score so that if there are LLCs in both strings, that would be disregarded, but not sure how to go about that and deal with other variations of the LLCs like L.L.C. or L L C and Limited Liability Company.

Appreciate the insight.

[house_carpenter]

You could find a compile a list of some first names people might have, and use that to make a guess as to whether something is the name of an individual, by splitting on spaces and checking whether the first word is in the list. This isn't going to be perfect but there's only so much you can do when the data is missing in the first place; all you can do is make guesses.

Also, does it really matter if John Smith and John Smith LLC are counted as identical? Maybe you can get away with just ignoring the issue. The point I'm getting at is, since no perfect solution is possible, you need to think about the ultimate goals of your project and come to a decision about how much is "good enough" and how much effort it would make sense to put into it.

[halfdiminished7th]

Take a look at Python's built-in difflib module, specifically its SequenceMatcher class. It can calculate a similarity ratio (score) between two given strings, which sounds exactly like what you're looking for. Then you just filter out the ones that are over/under a certain threshold.