Running Entity Resolution on Databricks With the Zingg App

Most data about people, companies, or products is scattered across systems and never quite agrees with itself. Zingg resolves it into one accurate record, running natively on your own data, with no data leaving your Databricks environment.

We are launching the Zingg Databricks app - an end to end pipeline for entity resolution. This app deploys Zingg Community on Databricks within your chosen workspace. All data and config - input and resolved output, Zingg configuration and models say in the user workspace. The app installs Zingg and brings the Zingg Python notebooks which can be customised by the end user.
‍

Prerequisites

• A Databricks workspace enabled for Unity Catalog.
• A single user cluster running Databricks Runtime 16.4 LTS, either existing or created for this purpose.
• The service principal must be granted access to the target cluster to manage it, and Can Edit permission on the destination Workspace folder to write notebooks into it.

The app authenticates as its own Databricks service principal for every action it performs: listing clusters, checking or installing the Zingg JAR, starting or restarting the selected cluster, and writing the bundled notebooks into your Workspace folder.

Installing the app involves the following steps

• Select a cluster: pick any existing single user cluster visible to you, and see its Spark version and access mode.
• Manage the Zingg JAR: check whether the bundled Zingg JAR (version 0.7.0) is already installed on that cluster, or install it, and start or restart the cluster as needed for the JAR to load.
• Get the notebooks: deploy the app's four bundled notebooks together into a Workspace folder you choose.
• Open a notebook: get a direct link to each deployed notebook, to attach to your cluster and run.

The bundled notebooks

The app comes with 4 Zingg notebooks. The notebooks are modularised to perform the following activities whent he user run them.

1. Setup: environment, paths, and field/matching configuration.
2. Label Training Data: interactive labeling of candidate record pairs as matches or non-matches.
3. Generate Document: produces a readable summary of your labelled data.
4. Train and Match: trains the model and runs matching, producing the resolved output table.

Each notebook refers to the first notebook so that the configuration can be reused. All four notebooks are deployed together into the same folder. You can edit the input and output table names, field definitions, and match types in notebook 1. Each notebook contains detailed instructions. You run all four notebooks directly, in order, on your selected cluster, entirely outside this app once they are deployed.

Architecture

The diagram below shows how the app interacts with your Databricks workspace. The app runs entirely inside your own workspace, on your own cluster compute. It only performs setup actions, listing clusters, installing the bundled Zingg JAR, starting or restarting the selected cluster, and copying the app's own bundled notebook files into your Workspace. All actual entity resolution work, including labeling, training, matching, and storing results, happens later when you run the deployed notebooks yourself, entirely on your own compute and against your own data. The app does not transmit any business data or match results externally at any point.

Setup

1. Deploy the app into the target Databricks workspace, using the included app.yaml, requirements.txt, and application source.
2. In Get Started, select the target cluster, then install the bundled Zingg JAR if it is not already present, restarting the cluster if prompted.
3. Choose a Workspace folder and click Get notebooks to deploy all four bundled notebooks into it together.

‍

Running the App

1. Open notebook 1 (Setup) first and edit it for your own tables, fields, and match types, then run all four notebooks in order on the selected cluster. Refer to the Databricks Guide for the details of the notebooks.

Support and community

Documentation: docs.zingg.ai
Community: Zingg Slack
For Zingg Enterprise, including persistent Zingg IDs and incremental matching for live entity graphs, contact us.

‍