Product
Solutions
Resources
About Us
Use Cases
Contact Us
.png)
Part 1 of 5. ← Series index · Part 2: The Noise Problem →
Every fuzzy matching project starts the same way. You reach for Levenshtein distance, compute a similarity score, pick a threshold, and call it done. For a small dataset with reasonably clean data, this works. Then you run it on your actual production data and discover that the world is messier than your sample suggested.
This first post examines why a single string similarity metric is structurally insufficient for real-world entity matching — and what a more complete approach looks like.
Before criticizing the standard approach, it's worth being clear about what it does well.
Levenshtein distance measures the minimum number of single-character edits — insertions, deletions, substitutions — needed to transform one string into another. It's intuitive, well-understood, and fast. For catching straightforward typos ("Micheal" → "Michael", "Smtih" → "Smith"), it's exactly the right tool.
Jaro-Winkler extends this with weighting that favors prefix matches and is more forgiving of character transpositions. It was designed specifically for short strings like personal names, and outperforms Levenshtein on that task in most benchmarks.
For small datasets, relatively clean data, and a single field to compare, a well-tuned Levenshtein or Jaro-Winkler threshold gets you surprisingly far. If you're checking whether a user-entered name matches a reference list of a few thousand entries, this is entirely sufficient.
The problems emerge when you go beyond any of those conditions.
Real-world name data doesn't fail to match in one way. It fails in at least six distinct ways — and each requires a different kind of treatment.
Typos — "Micheal" instead of "Michael", "Smyth" instead of "Smith". Character-level edit distance handles this well. This is the only failure mode that simple fuzzy matching was actually designed for.
Abbreviations — "J. Smith" vs "John Smith", "Wm. Jones" vs "William Jones". Levenshtein scores these as highly dissimilar strings — the edit distance between "J." and "John" is large — even though abbreviation is a completely standard naming convention. Any dataset assembled from multiple sources will be full of this.
Phonetic variation — "Jon" vs "John", "Catherine" vs "Katherine" vs "Kathryn", "Smith" vs "Smyth". Same sound, different spelling. Character-level metrics see these as different strings. Phonetic algorithms like Soundex, Metaphone, or Double Metaphone encode how a string sounds rather than how it's spelled, which makes these trivially equivalent.
Nicknames — "Bob" vs "Robert", "Bill" vs "William", "Liz" vs "Elizabeth", "Peggy" vs "Margaret". These have essentially no character overlap. No string similarity metric — not Levenshtein, not Jaro-Winkler, not any phonetic algorithm — will connect them. The only way to handle nicknames is an external lookup table mapping informal names to their formal equivalents.
Case and punctuation — "JOHN SMITH" vs "john smith" vs "John Smith." These are trivially the same string after normalization, but normalization is frequently skipped in pipelines assembled from multiple source systems with different conventions.
Language and script variation — "Müller" vs "Muller", "Søren" vs "Soren", or names written in different scripts entirely (a name in Arabic script vs its transliteration to Latin). Before any character-level comparison can happen, transliteration logic has to normalize the representation.
The key insight: each failure mode requires a different algorithm. Typos → edit distance. Phonetic variation → phonetic encoding. Nicknames → lookup tables. Script variation → transliteration. No single metric covers all of them, and no threshold tuning on a single metric compensates for the ones it doesn't cover.
The correct response is to stop looking for a single best metric and instead compose multiple signals — each handling a different class of variation — and let a classifier weigh them together.
This is the architecture that every production entity resolution system eventually converges on. As the Zingg scoring documentation describes it: "No individual feature is perfect, but the whole is greater than the sum of its parts." Zingg computes multiple features per field — string lengths and their differences, character differences, which characters actually differed — and feeds them to a classifier that learns which combination of signals predicts a true match for your specific data.
Critically, this means the matching behavior is learned from your data rather than hand-tuned. Different datasets have different patterns of variation. A medical records database will have different name conventions than a sports league fan database. A global enterprise CRM will have different script challenges than a domestic retailer. A model trained on labeled pairs from your data captures these patterns; a generic threshold on a generic metric does not.
In practice, multi-signal matching requires expressing not just how similar two records are across all fields, but what kind of similarity applies to each field.
An email address should not be compared the same way as a first name. A postcode should not be compared the same way as an address line. A country code should not be compared the same way as a company name. Each field has different conventions, different patterns of variation, and different tolerance for differences.
Zingg expresses this through field-level match types in the configuration:
[ { "fieldName": "firstName", "matchType": "FUZZY" }, { "fieldName": "lastName", "matchType": "FUZZY" }, { "fieldName": "email", "matchType": "EMAIL" }, { "fieldName": "postcode", "matchType": "PINCODE" }, { "fieldName": "country", "matchType": "EXACT" }, { "fieldName": "streetName", "matchType": "ONLY_ALPHABETS_FUZZY" }, { "fieldName": "streetNo", "matchType": "NUMERIC" }, { "fieldName": "notes", "matchType": "TEXT" }]
What each of these means in practice:
FUZZY — broad matching that handles typos, abbreviations, and other surface variations. The default for name fields. Uses a combination of character-level and token-level features.
EMAIL — matches only the local part before the @ character. Handles the common case where the same person has separate work and personal email addresses at different domains, or where the same person has multiple aliases at the same domain.
PINCODE — matches postal codes in different formats, like 12345 against 12345-6789. Handles the common inconsistency between 5-digit and ZIP+4 formats.
EXACT — no tolerance for variation. Use for categorical fields like country codes, ISO identifiers, or boolean flags where any difference is meaningful.
ONLY_ALPHABETS_FUZZY — strips numeric characters before fuzzy comparison. Useful for address lines where you want to match the street name but handle the street number separately via a NUMERIC field. Avoids the situation where "123 Main St" scores poorly against "456 Main St" because the numbers differ, even though they're clearly different addresses.
NUMERIC — extracts numbers from strings and compares how many are shared. Good for apartment numbers, floor numbers, and other numeric identifiers embedded in text fields.
TEXT — compares word overlap between two strings. Better for longer descriptive fields where character-level comparison is less meaningful.
NULL_OR_BLANK — by default Zingg treats null values as potential matches (since a missing value doesn't prove records are different). Adding this match type alongside FUZZY teaches the model to treat nulls as a distinct signal rather than an unknown.
The match types above all operate on the data as it exists. MAPPING is different: it uses a lookup file to define equivalences that no algorithm could infer from character patterns alone.
{ "fieldName": "firstName", "matchType": "MAPPING_(nicknames.csv)"}Where nicknames.csv contains rows like:Bob, RobertBill, WilliamLiz, ElizabethPeggy, Margaret
This handles the nickname problem that was otherwise unsolvable. The same mechanism works for company name abbreviations ("IBM" → "International Business Machines"), gender abbreviations ("M" / "0" / "Male"), honorifics, and any other domain-specific mapping where you have the knowledge but the algorithm doesn't.
The MAPPING match type is part of Zingg Enterprise, reflecting that maintaining and applying these lookup tables at scale is an operational concern beyond the core matching algorithm.
When you configure multi-signal matching with field-appropriate match types, you're not just improving accuracy on a few edge cases. You're changing the structural properties of your matching system:
The combined effect is a substantial improvement in recall — the proportion of true matches your system finds — without sacrificing precision. The classifier that weighs these signals together learns from your labeled data which combinations are predictive of a true match in your specific domain.
This is the first layer of a production fuzzy matching system. The next post covers the second: the noise in your data that actively corrupts your signals, and how to remove it before comparison.
Up next: Part 2 — The Noise Problem: Stopwords, Normalization, and Domain Tuning
← Series index
