Billion Scale Deduplication using Approximate Nearest Neighbours| Idan Richman Goshen, Sr Ds@Lusha

At Lusha we are dealing with contacts profiles, lots of contacts profiles. It is by nature messy, and a single entity can have several representations in this type of data. In addition to the time and money spent moving messy data through the various pipelines, it is difficult to search in, not to mention the valuable information lost in the process. It would be ideal if we could merge all records of the same entity, even if they differ slightly (“Alagra Jones”, “Alagra Smith-Jones”). Comparing combinations of all pairs is possible on a small scale, but impossible when dealing with billions of records.
A set of algorithms known as approximate nearest neighbours is becoming more popular for solving such challenges and allowing the use of text-embeddings and clustering at large scales.
This talk will offer a brief overview of ANN algorithms and demonstrate how we can apply them to get a reasonable size subset of candidates, which we can then pass into a classifier for a match/no-match outcome. I’ll demonstrate how we handle such a task at scale, how we evaluate the two steps, and the tools we use.