This is a five-part series on building production-grade entity resolution systems, from core matching techniques to operating a live identity graph.
Part 1: Why One Similarity Metric Is Never Enough
Part 2 :The Noise Problem: Stopwords, Normalization, and Domain Tuning
Part 3: Blocking: Making Billion-Record Matching Tractable
Part 4: Thresholds, Scores, and Active Learning
Part 5: Incremental Flow and Living Clusters
Every entity resolution project starts the same way. You reach for Levenshtein distance, compute a similarity score, pick a threshold, and call it done. For a small dataset with reasonably clean data, this works. Then you run it on your actual production data and discover that the world is messier than your sample suggested.
This first post examines why a single string similarity metric is structurally insufficient for real-world entity matching, and what a more complete approach looks like.
Before criticizing the standard approach, it's worth being clear about what it does well.
Levenshtein distance measures the minimum number of single-character edits, insertions, deletions, substitutions, needed to transform one string into another. It's intuitive, well-understood, and fast. For catching straightforward typos ("Micheal" → "Michael", "Smtih" → "Smith"), it's exactly the right tool.
Jaro-Winkler extends this with weighting that favors prefix matches and is more forgiving of character transpositions. It was designed specifically for short strings like personal names, and outperforms Levenshtein on that task in most benchmarks.
For small datasets, relatively clean data, and a single field to compare, a well-tuned Levenshtein or Jaro-Winkler threshold gets you surprisingly far. If you're checking whether a user-entered name matches a reference list of a few thousand entries, this is entirely sufficient.
The problems emerge when you go beyond any of those conditions.
Real-world name data doesn't fail to match in one way. It fails in at least six distinct ways, and each requires a different kind of treatment.
Typos
"Micheal" instead of "Michael", "Smyth" instead of "Smith". These are the cases Levenshtein handles well. Small edit distances, easily caught with a low threshold.
Abbreviations
"International Business Machines" vs "IBM". Edit distance between these strings is enormous, but they're the same entity. No edit-distance metric handles this without domain knowledge or alias tables.
Transpositions
"John Smith" vs "Smith John". Token order matters for substring metrics, even though it doesn't matter for entity identity. Jaro-Winkler handles character transpositions within tokens but not token-level reordering.
Missing tokens
"Sonal Goyal" vs "S Goyal". The first token is abbreviated to an initial. The string similarity is low, but the entities are the same. You need token-level comparison, not string-level.
Extra tokens
"Zingg Labs Inc" vs "Zingg". One string contains legal suffixes or qualifiers that the other omits. Without stopword handling, these tokens inflate the distance.
Phonetic variants
"Katherine" vs "Catherine", "Müller" vs "Mueller". These require phonetic encoding (Soundex, Metaphone, or similar) to match, not edit distance.
A single metric covers at most two or three of these cases well. The other cases require different approaches. This is why real-world entity matching cannot rely on a single similarity function.
A robust entity resolution system uses multiple similarity functions and combines their outputs into a composite score. The combination is what matters, not which specific metrics you choose, but how they're weighted and aggregated.
Zingg applies multiple string similarity metrics per field, Jaro-Winkler, Levenshtein, cosine similarity on token sets, phonetic similarity, and learns the appropriate weight for each metric per field from labeled training data. A field like "company name" gets different metric weights than a field like "email address", because the kinds of variation are structurally different.
This learned weighting is what distinguishes an ML-based approach from a rule-based one. You don't have to decide in advance that Jaro-Winkler should count for 60% of the name score. The model learns the right combination from examples of matching and non-matching pairs in your actual data.
Multi-metric scoring happens at the field level. But the final match decision is at the record level, two records either represent the same entity or they don't.
The translation from field-level scores to a record-level decision introduces additional complexity. Fields have different discriminative power. Email address is nearly deterministic: if it matches, records almost certainly represent the same entity. First name alone is not: two records with matching first names could easily be different people.
A composite record-level score needs to weight fields by their discriminative power, not just average them. A high email match should dominate a partial name match. A low email match shouldn't automatically disqualify records that match strongly on name, address, and phone.
Zingg's training process learns these field-level weights from labeled pairs in your data, rather than requiring them to be specified manually. The result is a match classifier that's calibrated to the actual distribution of your specific dataset, not to some generic assumption about how names and addresses behave.
Multi-metric comparison is computationally expensive. If you're computing Jaro-Winkler, Levenshtein, cosine, and phonetic similarity for every field, on every pair of records, the cost per comparison is significant. And if you're comparing every record to every other record, the number of pairs grows quadratically with dataset size.
For one million records, brute-force comparison produces 500 billion pairs. Even at microseconds per comparison, this is computationally impossible in any practical timeframe.
This is the blocking problem, and it's the subject of Part 3 of this series. Before multi-metric comparison can be applied, the comparison space has to be reduced to a manageable set of candidate pairs, without losing true matches in the process.
This post has focused on why a single similarity metric is insufficient and what a more complete field-level approach looks like. But comparison is only meaningful on clean data. Before any matching can work reliably, you have to deal with the noise in your inputs: stopwords, normalization inconsistencies, domain-specific patterns that distort similarity scores.
That's the subject of Part 2: The Noise Problem — Stopwords, Normalization, and Domain Tuning.
.png)
