Open Source Entity Resolution
Zingg finds the records that belong to the same customer, supplier or product, across systems and typos, with AI that learns from a few pairs you label. Zingg is warehouse native and free under AGPL v3.0. Use it from Python or JSON on the Spark inside your data platform, connect it to Snowflake, Databricks, BigQuery and more, and scale to millions of records.
Entity Resolution In Python
pip install zingg and write your fields and data in a few lines of Python, then label, train and match. Prefer config? The same runs from a JSON file.
Install
pip install zingg on the Spark you already use, in a notebook or a job.
Describe Your Fields
One FieldDefinition per column, with a match type such as FUZZY, EXACT or EMAIL.
Label A Few Pairs
Run findTrainingData and label, and answer yes, no or not sure for a few dozen pairs.
Train And Match
Train on your labels, then match. Every record gets a Z_CLUSTER ID shared with its duplicates.
pip install zingg from zingg.client import *
from zingg.pipes import *
args = Arguments()
fname = FieldDefinition("fname", "string", MatchType.FUZZY)
email = FieldDefinition("email", "string", MatchType.EMAIL)
args.setFieldDefinition([fname, email])
args.setModelId("customers")
args.setZinggDir("models")
customers = Pipe("customers", "delta")
customers.addProperty("location", "/lake/crm/customers")
args.setData(customers)
resolved = Pipe("resolved", "delta")
resolved.addProperty("location", "/lake/crm/resolved")
args.setOutput(resolved)
options = ClientOptions([ClientOptions.PHASE, "match"])
Zingg(args, options).initAndExecute()docker run -it zingg/zingg:0.7.0 bashAI Built For Entity Resolution
Zingg learns two models from your data: a blocking model that decides what to compare, and a similarity model that decides what matches.
Learns From A Few Labels
An interactive labeler picks the pairs it is least sure about, so a small, frugal training set is enough for an accurate model.
Learned Blocking
Zingg learns which records are worth comparing, so it compares 0.05 to 1% of all possible pairs and scales to millions of records.
A Similarity Model
A classifier trained on your labels scores the pairs inside each block, catching typos, abbreviations and reordered names.
Any Entity, Many Languages
Customers, patients, suppliers, products or households, in English, Chinese, Thai, Japanese, Hindi and other languages.
Flexible And Warehouse Native
Run Zingg on the Spark you already have and resolve data wherever it sits: Snowflake, Databricks, BigQuery, Redshift, databases or files. It works inside your data platform, on your compute, so nothing is copied out to another service.
On Your Compute
Zingg runs on the Spark in your Databricks, Microsoft Fabric, Google Cloud or AWS account, or your own cluster, next to your data.
Warehouse nativeConnects To Snowflake
Read Snowflake tables and write resolved records back to Snowflake through the Snowflake Spark connector. Zingg Enterprise also runs natively inside Snowflake.
Zingg on SnowflakeYour Tables, In Place
Zingg reads your Snowflake, BigQuery, Redshift and Delta tables, JDBC databases and files in cloud storage where they are, and writes the results back beside them.
Every connectorPython Or JSON
Describe your fields and data in Python with pip install zingg, or in a JSON config. Both run the same phases.
Python APIStart Open, Grow Into Production
Your labels, field definitions and pipelines carry over when you move to Zingg Enterprise.
| Zingg Community | Zingg Enterprise | |
|---|---|---|
| Price | Free, AGPL v3.0 | Priced per deployment |
| Matching | Probabilistic AI matching on your labels | Probabilistic and deterministic matching in one flow |
| Identifier | Z_CLUSTER per run | Persistent ZINGG_ID across runs, reassigned through migrations |
| Updates | Rerun match on the full data | Incremental runs on new, changed and deleted records |
| Names | Stopwords | Nicknames, company name dictionaries, standardize and match |
| Insight | Not included | Explainability, match statistics, model diff |
| Snowflake | Through the Snowflake Spark connector | Also native inside Snowflake |
| Support | The community on Slack and GitHub | Enterprise-grade support and bug fixes |
Built In The Open, With Data Teams
Ask questions on Slack, report bugs and request features on GitHub, and meet other practitioners at Zingg meetups.
Join SlackOpen Source Questions
Is Zingg really open source?
Yes. Zingg Community is on GitHub under the GNU Affero General Public License v3.0. You can read, run, change and redistribute the code.
Can I use Zingg Community at work?
Yes. AGPL allows unrestricted use by end users, solution builders and partners. If you change Zingg and offer it to others over a network, AGPL asks you to share those changes. Need a different license? Talk to us.
Does Zingg Community work with Snowflake?
Yes. Zingg Community reads Snowflake tables and writes the resolved records back to Snowflake through the Snowflake Spark connector, from Python or JSON. Zingg Enterprise can also run natively inside your Snowflake account.
How many records can Zingg Community handle?
Millions. Learned blocking means Zingg compares a small fraction of all possible pairs, and it runs on Spark, so you add compute as the data grows.
How much training data do I need?
Usually a few dozen labeled pairs to start. The labeler shows the pairs the model is least sure of, so each label teaches it the most.
Where does Zingg Community run?
Inside your data platform. Zingg is warehouse native: it runs on Spark in Databricks, Microsoft Fabric, Google Cloud Dataproc, AWS EMR or Glue, or your own cluster, and reads and writes your tables in place. To try it, run Docker on a laptop.
What does Zingg Enterprise add?
A persistent ZINGG_ID, incremental runs, deterministic rules alongside AI matching, nicknames and dictionaries, explainability, model diff, native Snowflake runs and enterprise support.
How do I get help or contribute?
Ask on the Zingg Slack or open a GitHub issue. To contribute code, see the contributing guide in the Zingg docs.
Resolve Your First EntitiesToday
Get Zingg Community from GitHub, or talk to us about Zingg Enterprise for production.