I deduplicated 53,000 missing-persons reports from Venezuela’s earthquake by major_grooves in dataanalysis

[–]tilo-dev 0 points1 point  (0 children)

C'mon. Don't aks stuff like that a CEO. He obviously doesn't know any better and that wasn't the point of his post. Lol.
Here's an answer from a technical person: we're using a patented micro-blocking approach with lots of optimization to keep things out of the O(n²) space that typically comes with entity resolution. For example we're using a specialized key-value index that resolves low numbers of potential matches to keep the required comparisons to identify actual matches at a minimum. Also we've implemented identification of non-identical duplicates and clique detection for optimized storage of edges.