In 1969, Ivan Fellegi and Alan Sunter published A Theory for Record Linkage in the Journal of the American Statistical Association.[1] Their paper formalised what had been intuited but never rigorously described: a statistical decision framework for determining whether two records refer to the same real-world entity. For several decades, it was the most principled tool available.
The model operates by comparing pairs of records across shared fields, computing a likelihood ratio, the ratio of match probabilities (m-probabilities) to non-match probabilities (u-probabilities) and classifying pairs as links, non-links, or potential matches requiring clerical review. The parameters are typically estimated through the Expectation-Maximisation (EM) algorithm, introduced into this context by Winkler (1988) and Jaro (1989).[2]
The model's elegance is undeniable. Its longevity is a testament to the quality of the original thinking. And yet, that framework now carries significant structural limitations when applied to the realities of modern data engineering.
What follows is a rigorous, evidence-based critique of those limitations, drawing on peer-reviewed literature, government research, and empirical benchmarks, alongside a description of what a model built for today's world actually looks like.
A framework designed for datasets in the thousands, running on 1960s hardware, without labelled training data, now applied to billion-record cloud data platforms. The assumptions have not aged equally. To Keep up with that MDM is also Evolving
To critique the Fellegi-Sunter model fairly, it is necessary to be precise about its assumptions. The framework rests on three that are mathematically essential, each of which can and routinely does fail in practice.
First: the model assumes that matching fields are conditionally independent given true match status. That is, whether two records agree on postcode tells us nothing, beyond what we already know about match status, about whether they agree on street name, city, or any other correlated field.
Second: it assumes that parameters can be estimated without labelled training examples, relying entirely on the EM algorithm applied to observed agreement patterns.
Third: it classifies each pair using a fixed score threshold, set by the practitioner, above which pairs are declared matches.
All three assumptions fail systematically in real-world data. The academic literature has spent the last three decades documenting exactly how and when they fail, proposing extensions, and demonstrating empirically that machine learning approaches outperform the baseline model on every accuracy metric that matters.
The model's core mathematical result that the overall match weight is the sum of per-field weights is only valid if fields are independent conditional on match status. In the words of the International Journal of Epidemiology's landmark tutorial on probabilistic record linkage: "conditional independence between the linkage keys of interest appears to be a key assumption in the Fellegi and Sunter formulation. However, in practice this assumption is likely to be violated." [Harron, Goldstein & Dibben, IJE 2015]
The violation is not marginal. Consider a dataset where postcode, street name, and county are all matching fields. Agreement on postcode is strongly predictive of agreement on street name, they are not independent. When this dependency is ignored, the model double-counts evidence and produces biased parameter estimates.
A rigorous study published in the Journal of the Royal Statistical Society (2022) quantified the damage directly: "consequences of violating the conditional independence assumption include bias in parameter estimates from the model... with a misspecified conditional dependence structure, these models can yield worse performance." [Xu, Li & Grannis, JRSS-C 2022] The same paper found that existing methods for detecting and correcting conditional dependence, the correlation residual plot, the log-odds ratio check, the bivariate residual, all perform inadequately.
Winkler himself, a leading figure in the Fellegi-Sunter tradition, proposed methods as early as 1989 to adjust for "lack of independence in an application of the Fellegi-Sunter model", an acknowledgment from within the framework's own lineage that the assumption it rests on cannot be taken for granted.
Sources: JRSS-C 2022 | Int. J. Epidemiology 2015 | PubMed: Conditional Dependence
Because Fellegi-Sunter was designed to operate without labelled training data, its parameters are estimated using the EM algorithm on observed agreement patterns alone. This was a practical necessity in 1969. Today, it is an architectural choice with real accuracy costs.
The paradox at the heart of unsupervised parameter estimation is well documented: to estimate the model's parameters, you need to know which records match, but finding which records match is the entire problem you are trying to solve. The EM algorithm resolves this by treating match status as a latent variable and iterating until convergence. But convergence is only to a local maximum of the likelihood function, not a global optimum.
A 2018 study published by the U.S. Bureau of Labor Statistics compared Fellegi-Sunter directly against supervised classification methods: logistic regression, support vector machines, gradient boosting, and random forests across the same dataset and evaluation measures. The finding was unambiguous: "the probabilistic method of Fellegi-Sunter performs the worst amongst all other classification techniques for all measures." The paper attributed this directly to the fact that supervised methods "have more features to work with, particularly those that are specific to one data source but not the other. The Fellegi-Sunter method can only look at directly comparable features." [Mason, BLS 2018]
A 2024 ONS/BMC study evaluating England and Wales census linkage reached the same conclusion: supervised ML models achieved higher F1 scores than the EM baseline across every evaluation. [Sutton et al., BMC 2024]
Sources: BLS Research Paper 2018 | ONS Census Linkage Study 2024
The Fellegi-Sunter decision rule classifies record pairs into three regions: definite matches above an upper threshold, definite non-matches below a lower threshold, and a middle band of potential matches requiring clerical review. The critical question, where to set those thresholds, is not answered by the model itself. It is left entirely to the practitioner.
A 2024 study in the International Journal of Population Data Science described this directly: "probabilistic record linkage heavily relies on manual intervention for gray area resolutions. This makes PRL extremely time and resource intensive." Their evaluation of 12 million records found that human errors were introduced during manual review due to the subjective nature of threshold decisions. [Azimaee et al., IJPDS 2024]
The Brazilian dengue-stillbirth linkage study (BMC Medical Informatics) illustrated the sensitivity starkly: with a conservative threshold, sensitivity was 64.4% and PPV 92.5%; with a relaxed threshold, sensitivity rose to 69.1% but PPV collapsed to 68.4%. The same data, the same model, two reasonable threshold choices, wildly different downstream reliability.
Sources: IJPDS 2024 | BMC: Brazil Dengue Linkage | PLOS One: Cancer Registry 2023
The original Fellegi-Sunter formulation was built around binary comparison vectors: two records either agree on a field or they do not. The underlying framework remains poorly suited to continuous data, low-prevalence binary fields, and the mixed-type data that characterises almost every real-world dataset.
A 2022 paper in Computational Statistics and Data Analysis, applied to the French national health data system, documented the limitation precisely: "most probabilistic record linkage models only make use of simple binary or categorical comparison values even if the matching variables are continuous... the simple binary comparison method proposed by Fellegi and Sunter cannot distinguish the agreement of low prevalence values, which is much more informative than the agreement of high prevalence values." [Courtier-Arnoux et al., CSDA 2022]
A study in the Journal of Applied Statistics showed that the standard FS model "leads to identical classification of match status regardless of whether records agree on rare or common values", a distinction the model, in its standard form, cannot exploit. [Xu, Li & Grannis, J. Applied Stats 2022]
Missing data compounds this further. A 2022 JMIR study at Indiana University School of Medicine demonstrated that the standard convention, treating missing values as disagreements, measurably degrades matching accuracy across four real-world clinical use cases. [Li, Xu & Grannis, JMIR 2022]
Sources: CSDA: French Health System | J. Applied Stats | JMIR: Missing Data
At any real-world scale, comparing every record against every other is computationally impossible. The pairwise comparison space is quadratic: two datasets of 1 million records yield a trillion possible pairs. Blocking is the universal workaround but it has a consequence the Fellegi-Sunter framework cannot handle cleanly: it creates a biased sample.
"Whether implemented by blocking, filtering, or some other process, indexing creates a biased sample of record pairs by design. For model-based procedures (including Fellegi and Sunter) indexing therefore changes the interpretation of the recovered parameters." [Ventura et al., cited in Coleridge Initiative Big Data and Social Science]
A 2020 paper in the Journal of Official Statistics confirmed that standard blocking in the FS context "produces a large bias of EM-estimated parameters" and that "the probability that excluded pairs would be actually true-matches cannot be assessed through such methods." [Fortini et al., JOS 2020]
Harvard and Princeton researchers identified the same problem: "current implementations do not scale" and the model handles "missing data in ad-hoc ways" and "does not incorporate auxiliary information." [Enamorado, Fifield & Imai, 2018]
Sources: J. Official Statistics 2020 | Princeton/Harvard 2018
The critique above is not a critique of probabilistic thinking in general. It is a critique of a specific 1969 architecture applied without modification to problems it was not designed to solve. The question is not whether to use statistics, it is whether the statistical model should be learned from your data or imposed upon it.
Zingg takes the second path. It is a supervised machine learning system for entity resolution, built from the ground up for distributed compute, open-source auditability, and domain-specific accuracy.
Zingg trains a similarity model from labelled pairs drawn from your actual data. With as few as 30–40 annotated examples, the model learns the similarity thresholds and field weights that reflect the real structure of your matching problem, not a generic prior applied uniformly.
Zingg uses active learning to select the most informative pairs for human labelling. The labeller sees pairs where the model is most uncertain, maximising the information gained per labelled example. This is how Zingg reaches high accuracy with small training sets: it labels intelligently, not exhaustively.
Zingg first learns a blocking model that clusters near-similar records before similarity scoring begins. Typical Zingg comparisons cover just 0.05–1% of the full pairwise space, without the statistical bias that afflicts fixed blocking rules in the Fellegi-Sunter framework. The blocking model scales natively to hundreds of millions of records on Spark, Databricks, and AWS Glue.
Because Zingg's model is a classifier trained on labelled data rather than a latent-class model with independence constraints, correlated fields are handled naturally. The model learns from the full joint pattern of field agreements and disagreements, including dependencies that the Fellegi-Sunter framework must either ignore or correct for through expensive model extensions.
Zingg allows you to define custom similarity functions for the specific fields in your domain. Legal entity names, product codes, address formats, medical identifiers, and financial instrument references all match differently. Zingg lets you encode that domain knowledge explicitly.
Every component of Zingg- the blocking model, the similarity model, the training data is inspectable and auditable. In regulated industries, this is not optional. Zingg's decisions can be traced to the labelled data that shaped the model. There is no black box, no proprietary parameter store, no vendor to ask for explanations.
The Fellegi-Sunter model deserves its place in the history of data science. It was a genuine intellectual achievement at a time when nothing like it existed. But the conditions of modern data engineering are not the conditions of 1969. They are distributed, heterogeneous, regulated, and operating at scales that would have been unimaginable then.
The assumptions that made the model tractable on mid-century hardware: conditional independence, unsupervised estimation, binary field comparison, manual threshold setting are not minor implementation details. The research is clear: supervised machine learning approaches outperform the probabilistic baseline on accuracy. Learned blocking outperforms fixed blocking rules on both bias and scale. Active learning makes labelling tractable even for small teams. Open-source, auditable models are not just technically superior, in regulated industries, they are increasingly a requirement.
Key references:
Fellegi & Sunter (JASA 1969)
Harron et al. (IJE 2015)
Xu, Li & Grannis (JRSS-C 2022)
Mason (BLS 2018)
Sutton et al. (BMC 2024)
Azimaee et al. (IJPDS 2024)
Courtier-Arnoux et al. (CSDA 2022)
Fortini et al. (JOS 2020)
Enamorado, Fifield & Imai (2018)
Li, Xu & Grannis (JMIR 2022)
.png)
