Scale To Millions

Comparing Every Record With Every Other Does Not Scale

Without a unique identifier, every record could match any other. Zingg takes care of scale for you: as you label pairs, it learns which records are worth comparing, and matching runs on millions of records on the platform you already have.

The All-Pairs Problem
Every record against every other1,000,000 records
0.3125M1.2550M5100M0Comparisons, quadrillionsRecordsIf work grew with recordsEvery pair
499,999,500,000

comparisons

About 14 hours at 10 million comparisons a second

10× the records means 100× the comparisons. Hover or drag along the chart to read any point.

Comparisons grow with the square of your records: ten times the records means a hundred times the work.

How Blocking Works

Learned While You Label, Nothing To Configure

The pairs you label to train matching also train the blocking model, automatically. There are no blocking keys to design, tune or maintain.

The Blocking Tree
Pick a record8 records

john doe goes j, then new, and is compared with johnh d oe and jon doe only.

8 records: 28 possible pairs, 4 comparedno setup
Blocking model, learned while you label
first1char(firstname)
j
first3char(city)
new
john doejohnh d oejon doe3 comparisons
san
john smithnone
m
last2char(lastname)
nn
mary annmarry ann1 comparison
es
mary jonesnone
t
tim chennone

Scale is built in: Zingg learns the blocking model while you label, so there is nothing to set up.

Automatic

Learned As You Label

Every Yes and No you give teaches Zingg which records belong together. From the same answers it builds the blocking model, so scale needs no extra work from you.

A tree, not one key

Different Fields On Different Branches

At every node of the blocking tree, Zingg picks the field and function that keep your labeled matches together. Each branch can split on a different combination, so near-similar records land in the same block and the cartesian join breaks.

firstnameWhich function keeps matches together?
labeled matchfirst1charlast1char
john / johnhj jn h
mary / marrym my y
last1char splits john from johnh, so first1char is chosenno match lost
Real Runs

From Thousands To Tens Of Millions

Run time depends on the number of records, the number of fields compared and how many duplicates there really are. These runs give a sense of the hardware you need.

120Krecords, Febrl sample data5 minutes4 cores, 10 GB RAM, local Spark
5Mrecords, North Carolina votersAbout 4 hours4 cores, 10 GB RAM, local Spark
9Mrecords, 3 fields: first name, last name, email45 minutesAWS m5.24xlarge, 96 cores, 384 GB RAM
80Mrecords, 8 to 10 fields, user reported, no tuningUnder 2 hours1 driver (32 cores) and 8 workers (64 cores)
Runs Where Your Data Lives

Scale With The Platform You Already Have

Up to a few million records, a single machine in Spark local mode is often enough. Beyond that, Zingg runs as a native workload on your Spark cluster or in Snowflake, and scales with it.

DatabricksSnowflakeMicrosoft FabricGCPAWS
Stay Fast As You Grow

Check, Tune And Run Only What Changed

Verify Blocking FirstEnterprise

Run verifyBlocking before a large match to see the record count per block and samples from each, and add labels if blocks are too big.

Resolve Only What ChangedEnterprise

Incremental runs match new and updated records against existing clusters instead of reprocessing all your data.

Incremental flow
FAQ

Scaling Questions

How many records can Zingg resolve?

Zingg has been run on tens of millions of records. In Zingg's own benchmarks, 9 million records with three fields took 45 minutes on a 96-core machine, and a user reported 80 million records with 8 to 10 fields in under 2 hours on a driver and 8 workers.

Do I need a Spark cluster?

Not for smaller data. With up to a few million records it can be easier to run Zingg on a single machine in Spark local mode. Larger data runs on a Spark cluster on Databricks, Microsoft Fabric, GCP or AWS, or natively in Snowflake.

How does Zingg avoid comparing every pair of records?

It learns a blocking model, a tree of fields and functions, from the pairs you label. Each node picks the field and function that keep your labeled matches together, different branches can split on different fields, and only records that end up in the same block are compared.

Do I need to configure blocking?

No. The blocking model is learned automatically from the same pairs you label to train matching, so there are no blocking keys to write or tune.

What affects how long a run takes?

The number of records, the number of fields being compared and the actual number of duplicates. Spark partitions matter too: keep numPartitions at 20 to 30 times the number of cores.

Can I check blocking before a large run?

Yes. In Zingg Enterprise, the verifyBlocking phase shows the record count per block and sample records from each, so you can see whether you need more labels before matching everything.

Resolve Millions Of RecordsOn Your Platform

Start with open source today, or talk to us about running Zingg Enterprise on your data.