I have been working in data for a long time, both on the warehouse and the datalake sides of things. As a boutique consultancy, we were often tasked with integrating data from multiple internal systems and building dashboards and models on top. Sometimes the data needed enrichment too. Straightforward, in theory.
It was back in 2013 that we got stuck on a project and failed miserably in our first attempt. The work needed us to integrate customer records from three different Oracle systems. No common unique identifiers. No standard definition of who a customer even was. Lots of variation in the data. Existing tools didn’t fit our stack, cost a fortune, and looked like black boxes with a long learning curve.
How tough could it be? — the programmer in me thought.
I took a sample of 25,000 records and wrote a small algorithm to match them. It crashed my computer. I had no background in entity resolution at that point and didn’t understand how hard it is to compare everything with everything. I rewrote it, threw some hardware at it, moved to Hadoop, then Spark. We delivered something that saved us. But I wasn’t too proud.
A few weeks later we stumbled into the same problem on a different project, enriching internal customer data with third-party data sourced externally. This time I read up. Arxiv was my friend! Top ML researchers were working on record linkage. Most conferences had a paper or two on it. It was an important AND a hard problem. Clearly I had a lot to catch up on.
That project worked. I wrote something that scaled decently and handled different types of entities. I talked about it at Spark Summit. Then got on with other work, as is usual in data consultancies.
But the problem never quite left me.
The moment I knew this had to exist
In 2015, a Hong Kong-based event management company found me through my Spark Summit talk. They were deduplicating customer records manually. It was a big drain on their time and resources, and it was wrecking all the customer analytics they wanted to do. They didn’t want to buy an expensive MDM or identity resolution tool. The open source libraries they had tried hadn’t worked.
I was excited and slightly terrified. Their data was in Chinese and Japanese and English. They wanted something that would work up to a million records, larger than anything we’d done before.
One thing I learned that was eye-opening: they didn’t care about the algorithm. They just wanted the problem solved, in any way it could be solved, with a repeatable process. Thankfully they had some training data we could use. It was a stretch. But it worked. And I couldn’t be happier.
There were many things I wanted to build next algorithms, scale, better generalization. But growing work and other priorities pushed me in another direction. I kept the conversations going, kept speaking about it, kept hearing from others running into the same wall. But I didn’t build.
Why now
Things changed. Thousands of companies moved to Snowflake. Databricks rose and rose (and raised :-) ). The modern data stack arrived, proper practices around extraction, loading, transformation. Analytics became a profession. The infrastructure finally caught up.
And the problem was still there, completely unsolved.
How can you build Customer 360 if the data isn’t unified? How can you do customer lifetime value, segmentation, or personalization when you have five records of the same customer in the same table? What is AML if the links between customers and households aren’t established? What about all those company names showing up in every vendor system and B2B enterprise, never reconciled?
When I paused to ask myself what I actually wanted to work on, this was at the top of the list. I just saw it everywhere.So in 2020, I started again from scratch and wrote Zingg from the ground up.
Why open source
I have been consuming open source software all my life right from Java in 1998. It felt right to give something back. But more than that, I genuinely believe open source Zingg is more powerful than a closed version because then it’s up to the community to find uses I haven’t seen and couldn’t anticipate. That’s not just a philosophy. In the years since the first commit, Zingg has been applied to patient matching in healthcare, anti-money laundering in financial services, supplier deduplication in procurement, and fan identity at the Canadian Football League. None of those were on my roadmap in 2020.
It is also a promise to work closely with different people, to collaborate, to build in public. That promise has been kept.
Where we are now
Zingg runs natively on Databricks, Snowflake, Microsoft Fabric, BigQuery, AWS Glue, and Redshift. Customers like Fortnum & Mason, the Canadian Football League, and Redica Systems have built production identity graphs on it. Billions of records resolved. A persistent ZINGG_ID that downstream systems can rely on as data changes over time.
The modern data stack is still missing its identity layer. And We’re building it.
.png)
