Product
Solutions
Resources
About Us
Use Cases
Contact Us
.png)
From Levenshtein to living clusters — everything a data engineer needs to know to build fuzzy matching that actually works in production.
Every engineer who has worked with identity data has had the same experience. You start with a reasonable assumption — "how hard can it be to match these records?" — write something simple, and watch it work on the sample data. Then you run it on production data. And the education begins.
This series is the education, written down. It covers the five compounding problems that turn "simple string matching" into a genuine engineering challenge, and how to address each one. The examples and tooling references are drawn from Zingg, an open-source entity resolution framework built on Apache Spark — but the concepts apply to any fuzzy matching system you're building or evaluating.
The obvious starting point — Levenshtein, Jaro-Winkler, phonetic algorithms — and why each one captures some failure modes but misses others. How names actually vary in the real world (typos, abbreviations, phonetics, nicknames, scripts), and why the answer is composing multiple signals rather than picking the best single one. Includes a practical look at Zingg's field-level match types.
Read this if: You're wondering why your string similarity score keeps missing obvious matches or producing false positives.
Why common tokens like "Inc.", "Ltd.", "Street", "Mr." destroy your matching signal — and what to do about it. Covers stopword detection, domain-specific normalization, and the surprisingly large accuracy gains that come from preprocessing. Includes Zingg's recommend phase and the MAPPING match type for nicknames and abbreviation tables.
Read this if: Your company name or address matching is producing noisy results despite a reasonable similarity metric.
The O(n²) problem and why it matters. What blocking is, how learned blocking differs from hand-crafted blocking rules, the failure modes of bad blocking (silent false negatives), and how to verify your blocking model before committing to a full run. Real performance numbers: 120k records in 5 minutes, 80 million records in under 2 hours.
Read this if: Your matching job is too slow, or you're worried about records that should match but never get compared.
Why a single global threshold is almost never the right answer. How Zingg's scoring model works, what transitivity means for cluster formation, and how to interpret low-scoring matches correctly. The training data bottleneck — why you need labeled pairs, and how active learning makes it possible to build a good model from 30–50 examples rather than thousands. Real-world case studies from the CFL and Fortnum & Mason.
Read this if: You're tuning a threshold by hand and it keeps being wrong, or you don't know where to get training data.
The problem most teams don't anticipate until it breaks them: what happens when data changes. Why full reruns don't work at scale. The three hard incremental problems — new records joining clusters, updated records triggering reassignment, and cluster merge/unmerge. Why human feedback must survive incremental runs. The role of ZINGG_ID as a stable, durable entity identifier. And how to keep IDs stable across model upgrades and infrastructure migrations.
Read this if: Your static matching pipeline works but you're not sure how to keep it current, or you've had downstream systems break because entity IDs changed.
Primarily data engineers and analytics engineers who are building or evaluating a fuzzy matching or entity resolution system — whether from scratch, with an open-source framework, or as part of a broader data platform.
It's also useful for data architects thinking about identity resolution as part of an MDM, CDP, or data lakehouse design, and for technical product managers who need to understand why "deduplicate the database" is a more complex project than it sounds.
The series assumes you're comfortable reading code and SQL, but doesn't assume prior knowledge of entity resolution, information retrieval, or machine learning.
Each part stands alone — you can read them in any order depending on which problem you're facing right now. But they're designed to build on each other. The problems compound: multi-signal matching only helps if your blocking model doesn't miss the pairs you need to compare; good blocking only helps if your threshold and scoring are correctly calibrated; all of it only works if your incremental flow keeps the resolved state current and consistent over time.
The arc of the series is the arc most teams follow in practice: get matching working on a sample, then discover each of these problems in roughly the order they're written, and progressively build a more complete system.
